Does value of docs.count in elasticsearch also include value of docs.deleted?
up vote
1
down vote
favorite
I have an index with the following docs.count and docs.deleted.
docs.count = 2762694
docs.deleted = 2508162
Is docs.count the total number of documents in disk including the ones marked as deleted?
Or is the total number of documents in disk is docs.count + docs.deleted?
UPDATE: To clarify my concern, if docs.count also include the docs.deleted in it, then in the above example about 90% of the documents are deleted. But on the other hand the docs.deleted is not counted as part of docs.count and they are additional then only 47% of the documents are deleted. Based on this I can decide how important it is to use force merge.
elasticsearch asked Nov 22 at 10:38
Thomas
1,8942140
add a comment |
up vote
1
down vote
favorite
I have an index with the following docs.count and docs.deleted.
docs.count = 2762694
docs.deleted = 2508162
Is docs.count the total number of documents in disk including the ones marked as deleted?
Or is the total number of documents in disk is docs.count + docs.deleted?
UPDATE: To clarify my concern, if docs.count also include the docs.deleted in it, then in the above example about 90% of the documents are deleted. But on the other hand the docs.deleted is not counted as part of docs.count and they are additional then only 47% of the documents are deleted. Based on this I can decide how important it is to use force merge.
elasticsearch asked Nov 22 at 10:38
Thomas
1,8942140
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have an index with the following docs.count and docs.deleted.
docs.count = 2762694
docs.deleted = 2508162
Is docs.count the total number of documents in disk including the ones marked as deleted?
Or is the total number of documents in disk is docs.count + docs.deleted?
UPDATE: To clarify my concern, if docs.count also include the docs.deleted in it, then in the above example about 90% of the documents are deleted. But on the other hand the docs.deleted is not counted as part of docs.count and they are additional then only 47% of the documents are deleted. Based on this I can decide how important it is to use force merge.
elasticsearch asked Nov 22 at 10:38
Thomas
1,8942140
I have an index with the following docs.count and docs.deleted.
docs.count = 2762694
docs.deleted = 2508162
Is docs.count the total number of documents in disk including the ones marked as deleted?
Or is the total number of documents in disk is docs.count + docs.deleted?
UPDATE: To clarify my concern, if docs.count also include the docs.deleted in it, then in the above example about 90% of the documents are deleted. But on the other hand the docs.deleted is not counted as part of docs.count and they are additional then only 47% of the documents are deleted. Based on this I can decide how important it is to use force merge.
elasticsearch elasticsearch asked Nov 22 at 10:38
Thomas
1,8942140
asked Nov 22 at 10:38
Thomas
1,8942140
edited Nov 22 at 13:45
asked Nov 22 at 10:38
Thomas
1,8942140
asked Nov 22 at 10:38
Thomas
1,8942140
asked Nov 22 at 10:38
Thomas
1,8942140
1,8942140
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
2
down vote
accepted
The counts you see in the _cat/indices API response are counts at the Lucene level, so docs.count also includes hidden nested documents.
docs.deleted are indeed documents marked as deleted but not yet expunged.
If you want a real count of your documents you should use the _cat/count API which provides a clean count that indicates the number of live documents and does not include deleted documents which have not yet been cleaned up by the merge process.
answered Nov 22 at 10:52
Val
100k6130167
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
No it doesn't include that number. Also note thatdocs.deleteddoesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.
– Val
Nov 22 at 13:42
add a comment |
up vote
1
down vote
No.
The docs.count does not include the docs.deleted.
docs.deleted are the documents that are marked for deletion by elasticsearch.
I don't know the exact operation you are performing but the docs.deleted can be this high even when you update a lot of your documents. As elasticsearch doesn't update the document in-place but deletes the older one and replaces it with the newer one.
answered Nov 26 at 15:45
deep
682621
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53429060%2fdoes-value-of-docs-count-in-elasticsearch-also-include-value-of-docs-deleted%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
The counts you see in the _cat/indices API response are counts at the Lucene level, so docs.count also includes hidden nested documents.
docs.deleted are indeed documents marked as deleted but not yet expunged.
If you want a real count of your documents you should use the _cat/count API which provides a clean count that indicates the number of live documents and does not include deleted documents which have not yet been cleaned up by the merge process.
answered Nov 22 at 10:52
Val
100k6130167
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
No it doesn't include that number. Also note thatdocs.deleteddoesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.
– Val
Nov 22 at 13:42
add a comment |
up vote
2
down vote
accepted
The counts you see in the _cat/indices API response are counts at the Lucene level, so docs.count also includes hidden nested documents.
docs.deleted are indeed documents marked as deleted but not yet expunged.
If you want a real count of your documents you should use the _cat/count API which provides a clean count that indicates the number of live documents and does not include deleted documents which have not yet been cleaned up by the merge process.
answered Nov 22 at 10:52
Val
100k6130167
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
No it doesn't include that number. Also note thatdocs.deleteddoesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.
– Val
Nov 22 at 13:42
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
The counts you see in the _cat/indices API response are counts at the Lucene level, so docs.count also includes hidden nested documents.
docs.deleted are indeed documents marked as deleted but not yet expunged.
If you want a real count of your documents you should use the _cat/count API which provides a clean count that indicates the number of live documents and does not include deleted documents which have not yet been cleaned up by the merge process.
answered Nov 22 at 10:52
Val
100k6130167
The counts you see in the _cat/indices API response are counts at the Lucene level, so docs.count also includes hidden nested documents.
docs.deleted are indeed documents marked as deleted but not yet expunged.
If you want a real count of your documents you should use the _cat/count API which provides a clean count that indicates the number of live documents and does not include deleted documents which have not yet been cleaned up by the merge process.
answered Nov 22 at 10:52
Val
100k6130167
answered Nov 22 at 10:52
Val
100k6130167
answered Nov 22 at 10:52
Val
100k6130167
answered Nov 22 at 10:52
Val
100k6130167
100k6130167
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
No it doesn't include that number. Also note thatdocs.deleteddoesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.
– Val
Nov 22 at 13:42
add a comment |
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
No it doesn't include that number. Also note thatdocs.deleteddoesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.
– Val
Nov 22 at 13:42
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
yes, docs.count include hidden and nested document, but does it also contain deleted, specified by docs.deleted? I'll edit the question to make it clear.
– Thomas
Nov 22 at 13:40
1
1
No it doesn't include that number. Also note that
docs.deleted doesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.– Val
Nov 22 at 13:42
No it doesn't include that number. Also note that
docs.deleted doesn't necessarily mean that there are that many deleted documents, but the same document might have been updated/index/deleted several times.– Val
Nov 22 at 13:42
add a comment |
up vote
1
down vote
No.
The docs.count does not include the docs.deleted.
docs.deleted are the documents that are marked for deletion by elasticsearch.
I don't know the exact operation you are performing but the docs.deleted can be this high even when you update a lot of your documents. As elasticsearch doesn't update the document in-place but deletes the older one and replaces it with the newer one.
answered Nov 26 at 15:45
deep
682621
add a comment |
up vote
1
down vote
No.
The docs.count does not include the docs.deleted.
docs.deleted are the documents that are marked for deletion by elasticsearch.
I don't know the exact operation you are performing but the docs.deleted can be this high even when you update a lot of your documents. As elasticsearch doesn't update the document in-place but deletes the older one and replaces it with the newer one.
answered Nov 26 at 15:45
deep
682621
add a comment |
up vote
1
down vote
up vote
1
down vote
No.
The docs.count does not include the docs.deleted.
docs.deleted are the documents that are marked for deletion by elasticsearch.
I don't know the exact operation you are performing but the docs.deleted can be this high even when you update a lot of your documents. As elasticsearch doesn't update the document in-place but deletes the older one and replaces it with the newer one.
answered Nov 26 at 15:45
deep
682621
No.
The docs.count does not include the docs.deleted.
docs.deleted are the documents that are marked for deletion by elasticsearch.
I don't know the exact operation you are performing but the docs.deleted can be this high even when you update a lot of your documents. As elasticsearch doesn't update the document in-place but deletes the older one and replaces it with the newer one.
answered Nov 26 at 15:45
deep
682621
answered Nov 26 at 15:45
deep
682621
answered Nov 26 at 15:45
deep
682621
answered Nov 26 at 15:45
deep
682621
682621
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53429060%2fdoes-value-of-docs-count-in-elasticsearch-also-include-value-of-docs-deleted%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
