Elasticsearch - how to group by and count matches in an index
I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:
|____Type_____|__ Date_added __ |
| walking | 2018-11-27T00:00:00.000 |
| walking | 2018-11-26T00:00:00.000 |
| running | 2018-11-24T00:00:00.000 |
| running | 2018-11-25T00:00:00.000 |
| walking | 2018-11-27T04:00:00.000 |
I want to group by and count how many matches were found for the "type" field, in a certain range.
In SQL I would do something like this:
select type,
count(type)
from index
where date_added between '2018-11-20' and '2018-11-30'
group by type
I want to get something like this:
| type | count |
| running | 2 |
| walking | 3 |
I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders
.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?
java elasticsearch
add a comment |
I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:
|____Type_____|__ Date_added __ |
| walking | 2018-11-27T00:00:00.000 |
| walking | 2018-11-26T00:00:00.000 |
| running | 2018-11-24T00:00:00.000 |
| running | 2018-11-25T00:00:00.000 |
| walking | 2018-11-27T04:00:00.000 |
I want to group by and count how many matches were found for the "type" field, in a certain range.
In SQL I would do something like this:
select type,
count(type)
from index
where date_added between '2018-11-20' and '2018-11-30'
group by type
I want to get something like this:
| type | count |
| running | 2 |
| walking | 3 |
I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders
.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?
java elasticsearch
add a comment |
I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:
|____Type_____|__ Date_added __ |
| walking | 2018-11-27T00:00:00.000 |
| walking | 2018-11-26T00:00:00.000 |
| running | 2018-11-24T00:00:00.000 |
| running | 2018-11-25T00:00:00.000 |
| walking | 2018-11-27T04:00:00.000 |
I want to group by and count how many matches were found for the "type" field, in a certain range.
In SQL I would do something like this:
select type,
count(type)
from index
where date_added between '2018-11-20' and '2018-11-30'
group by type
I want to get something like this:
| type | count |
| running | 2 |
| walking | 3 |
I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders
.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?
java elasticsearch
I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:
|____Type_____|__ Date_added __ |
| walking | 2018-11-27T00:00:00.000 |
| walking | 2018-11-26T00:00:00.000 |
| running | 2018-11-24T00:00:00.000 |
| running | 2018-11-25T00:00:00.000 |
| walking | 2018-11-27T04:00:00.000 |
I want to group by and count how many matches were found for the "type" field, in a certain range.
In SQL I would do something like this:
select type,
count(type)
from index
where date_added between '2018-11-20' and '2018-11-30'
group by type
I want to get something like this:
| type | count |
| running | 2 |
| walking | 3 |
I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders
.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?
java elasticsearch
java elasticsearch
edited Nov 27 '18 at 13:37
KimioN42
asked Nov 27 '18 at 13:25
KimioN42KimioN42
84118
84118
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
That's a good start! Now you need to add a terms
aggregation to your query:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
// add these two lines
TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
sourceBuilder.aggregation(groupBy);
1
Ok, try withtype.keyword
instead oftype
as the field name. I've updated my answer
– Val
Nov 27 '18 at 14:36
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.type
is of typetext
whiletype.keyword
is of typekeyword
– Val
Nov 27 '18 at 14:48
1
have you triedsearchResponse.getAggregations().get("byType").getBuckets()
?
– Val
Nov 28 '18 at 17:22
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
|
show 6 more comments
After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:
Terms terms = searchResponse.getAggregations().get("byType");
Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
for (Bucket bucket : buckets) {
System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
}
This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:
Type: walking = Count(900)
Type: running = Count(1800)
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53500771%2felasticsearch-how-to-group-by-and-count-matches-in-an-index%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
That's a good start! Now you need to add a terms
aggregation to your query:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
// add these two lines
TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
sourceBuilder.aggregation(groupBy);
1
Ok, try withtype.keyword
instead oftype
as the field name. I've updated my answer
– Val
Nov 27 '18 at 14:36
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.type
is of typetext
whiletype.keyword
is of typekeyword
– Val
Nov 27 '18 at 14:48
1
have you triedsearchResponse.getAggregations().get("byType").getBuckets()
?
– Val
Nov 28 '18 at 17:22
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
|
show 6 more comments
That's a good start! Now you need to add a terms
aggregation to your query:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
// add these two lines
TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
sourceBuilder.aggregation(groupBy);
1
Ok, try withtype.keyword
instead oftype
as the field name. I've updated my answer
– Val
Nov 27 '18 at 14:36
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.type
is of typetext
whiletype.keyword
is of typekeyword
– Val
Nov 27 '18 at 14:48
1
have you triedsearchResponse.getAggregations().get("byType").getBuckets()
?
– Val
Nov 28 '18 at 17:22
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
|
show 6 more comments
That's a good start! Now you need to add a terms
aggregation to your query:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
// add these two lines
TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
sourceBuilder.aggregation(groupBy);
That's a good start! Now you need to add a terms
aggregation to your query:
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);
// add these two lines
TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
sourceBuilder.aggregation(groupBy);
edited Nov 27 '18 at 14:36
answered Nov 27 '18 at 13:39
ValVal
106k6145179
106k6145179
1
Ok, try withtype.keyword
instead oftype
as the field name. I've updated my answer
– Val
Nov 27 '18 at 14:36
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.type
is of typetext
whiletype.keyword
is of typekeyword
– Val
Nov 27 '18 at 14:48
1
have you triedsearchResponse.getAggregations().get("byType").getBuckets()
?
– Val
Nov 28 '18 at 17:22
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
|
show 6 more comments
1
Ok, try withtype.keyword
instead oftype
as the field name. I've updated my answer
– Val
Nov 27 '18 at 14:36
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.type
is of typetext
whiletype.keyword
is of typekeyword
– Val
Nov 27 '18 at 14:48
1
have you triedsearchResponse.getAggregations().get("byType").getBuckets()
?
– Val
Nov 28 '18 at 17:22
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
1
1
Ok, try with
type.keyword
instead of type
as the field name. I've updated my answer– Val
Nov 27 '18 at 14:36
Ok, try with
type.keyword
instead of type
as the field name. I've updated my answer– Val
Nov 27 '18 at 14:36
1
1
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.
type
is of type text
while type.keyword
is of type keyword
– Val
Nov 27 '18 at 14:48
Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field.
type
is of type text
while type.keyword
is of type keyword
– Val
Nov 27 '18 at 14:48
1
1
have you tried
searchResponse.getAggregations().get("byType").getBuckets()
?– Val
Nov 28 '18 at 17:22
have you tried
searchResponse.getAggregations().get("byType").getBuckets()
?– Val
Nov 28 '18 at 17:22
1
1
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…
– Val
Nov 28 '18 at 18:06
1
1
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
Ok, make sure to specify which version of ES and the REST client you're running
– Val
Nov 28 '18 at 19:11
|
show 6 more comments
After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:
Terms terms = searchResponse.getAggregations().get("byType");
Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
for (Bucket bucket : buckets) {
System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
}
This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:
Type: walking = Count(900)
Type: running = Count(1800)
add a comment |
After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:
Terms terms = searchResponse.getAggregations().get("byType");
Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
for (Bucket bucket : buckets) {
System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
}
This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:
Type: walking = Count(900)
Type: running = Count(1800)
add a comment |
After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:
Terms terms = searchResponse.getAggregations().get("byType");
Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
for (Bucket bucket : buckets) {
System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
}
This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:
Type: walking = Count(900)
Type: running = Count(1800)
After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:
Terms terms = searchResponse.getAggregations().get("byType");
Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
for (Bucket bucket : buckets) {
System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
}
This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:
Type: walking = Count(900)
Type: running = Count(1800)
answered Nov 28 '18 at 19:36
KimioN42KimioN42
84118
84118
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53500771%2felasticsearch-how-to-group-by-and-count-matches-in-an-index%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown