Using ReplacingMergeTree as an updatable table: how to delete?
I'm using ClickHouse for a "kind of updatable large (hundreds millions rows) table" with ReplacingMergeTree. I need to upsert by batch and do some non-aggregated select. It works fine.
Even though it's a bit of a hack, and far from optimal (I mean unlike Clickhouse for OLAP), it can scale quite well and still performs faster than systems more or less dedicated to this like HBase or RDBMs (for my needs).
I use a ReplacingMergeTree table with a key:
CREATE TABLE Things (Key Int32, ValueA Int32, ValueB Int32)
ENGINE = ReplacingMergeTree() ORDER BY Key
I upsert with:
INSERT INTO Things (Key,ValueA,ValueB) ...
And select with the "FINAL" modifier:
SELECT Key,ValueA,ValueB FROM Things FINAL WHERE ...
I can "delete" objects by using a column named "Killed". But from time to time, I need to clean "Killed" objects to prevent the table from growing endlessly.
The only method I found was recreating a new table and inserting in it non killed rows. Is there a smarter way?
clickhouse
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
add a comment |
I'm using ClickHouse for a "kind of updatable large (hundreds millions rows) table" with ReplacingMergeTree. I need to upsert by batch and do some non-aggregated select. It works fine.
Even though it's a bit of a hack, and far from optimal (I mean unlike Clickhouse for OLAP), it can scale quite well and still performs faster than systems more or less dedicated to this like HBase or RDBMs (for my needs).
I use a ReplacingMergeTree table with a key:
CREATE TABLE Things (Key Int32, ValueA Int32, ValueB Int32)
ENGINE = ReplacingMergeTree() ORDER BY Key
I upsert with:
INSERT INTO Things (Key,ValueA,ValueB) ...
And select with the "FINAL" modifier:
SELECT Key,ValueA,ValueB FROM Things FINAL WHERE ...
I can "delete" objects by using a column named "Killed". But from time to time, I need to clean "Killed" objects to prevent the table from growing endlessly.
The only method I found was recreating a new table and inserting in it non killed rows. Is there a smarter way?
clickhouse
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
add a comment |
I'm using ClickHouse for a "kind of updatable large (hundreds millions rows) table" with ReplacingMergeTree. I need to upsert by batch and do some non-aggregated select. It works fine.
Even though it's a bit of a hack, and far from optimal (I mean unlike Clickhouse for OLAP), it can scale quite well and still performs faster than systems more or less dedicated to this like HBase or RDBMs (for my needs).
I use a ReplacingMergeTree table with a key:
CREATE TABLE Things (Key Int32, ValueA Int32, ValueB Int32)
ENGINE = ReplacingMergeTree() ORDER BY Key
I upsert with:
INSERT INTO Things (Key,ValueA,ValueB) ...
And select with the "FINAL" modifier:
SELECT Key,ValueA,ValueB FROM Things FINAL WHERE ...
I can "delete" objects by using a column named "Killed". But from time to time, I need to clean "Killed" objects to prevent the table from growing endlessly.
The only method I found was recreating a new table and inserting in it non killed rows. Is there a smarter way?
clickhouse
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
I'm using ClickHouse for a "kind of updatable large (hundreds millions rows) table" with ReplacingMergeTree. I need to upsert by batch and do some non-aggregated select. It works fine.
Even though it's a bit of a hack, and far from optimal (I mean unlike Clickhouse for OLAP), it can scale quite well and still performs faster than systems more or less dedicated to this like HBase or RDBMs (for my needs).
I use a ReplacingMergeTree table with a key:
CREATE TABLE Things (Key Int32, ValueA Int32, ValueB Int32)
ENGINE = ReplacingMergeTree() ORDER BY Key
I upsert with:
INSERT INTO Things (Key,ValueA,ValueB) ...
And select with the "FINAL" modifier:
SELECT Key,ValueA,ValueB FROM Things FINAL WHERE ...
I can "delete" objects by using a column named "Killed". But from time to time, I need to clean "Killed" objects to prevent the table from growing endlessly.
The only method I found was recreating a new table and inserting in it non killed rows. Is there a smarter way?
clickhouse
clickhouse
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
edited Nov 28 '18 at 17:49
Benoit Sanchez
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
asked Nov 28 '18 at 17:09
Benoit SanchezBenoit Sanchez
296113
296113
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
ClickHouse supports DML operations in recent releases, so you don't need ReplacingMergeTree to tombstone records like that.
Checkout https://clickhouse.yandex/docs/en/query_language/alter/#mutations for more details.
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53524720%2fusing-replacingmergetree-as-an-updatable-table-how-to-delete%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
ClickHouse supports DML operations in recent releases, so you don't need ReplacingMergeTree to tombstone records like that.
Checkout https://clickhouse.yandex/docs/en/query_language/alter/#mutations for more details.
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
add a comment |
ClickHouse supports DML operations in recent releases, so you don't need ReplacingMergeTree to tombstone records like that.
Checkout https://clickhouse.yandex/docs/en/query_language/alter/#mutations for more details.
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
add a comment |
ClickHouse supports DML operations in recent releases, so you don't need ReplacingMergeTree to tombstone records like that.
Checkout https://clickhouse.yandex/docs/en/query_language/alter/#mutations for more details.
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
ClickHouse supports DML operations in recent releases, so you don't need ReplacingMergeTree to tombstone records like that.
Checkout https://clickhouse.yandex/docs/en/query_language/alter/#mutations for more details.
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
answered Nov 29 '18 at 1:46
AmosAmos
1,80831029
1,80831029
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
add a comment |
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
I do "small batch" updates (like 1000 records per batch). That's why I believe ReplacingMergeTree is better than muation. But thanks, for cleaning mutation seems to be a very good idea.
– Benoit Sanchez
Nov 29 '18 at 11:07
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
yeah, that's reasonable
– Amos
Nov 29 '18 at 12:08
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53524720%2fusing-replacingmergetree-as-an-updatable-table-how-to-delete%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
