Elasticsearch - how to group by and count matches in an index












0















I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:




|____Type_____|__ Date_added __ |



| walking | 2018-11-27T00:00:00.000 |



| walking | 2018-11-26T00:00:00.000 |



| running | 2018-11-24T00:00:00.000 |



| running | 2018-11-25T00:00:00.000 |



| walking | 2018-11-27T04:00:00.000 |




I want to group by and count how many matches were found for the "type" field, in a certain range.
In SQL I would do something like this:



 select type,  
count(type)
from index
where date_added between '2018-11-20' and '2018-11-30'
group by type


I want to get something like this:




| type | count |



| running | 2 |



| walking | 3 |




I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:



SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders
.boolQuery()
.must(QueryBuilders
.rangeQuery("date_added")
.from(start.getTime())
.to(end.getTime()))
)
);


How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?










share|improve this question





























    0















    I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:




    |____Type_____|__ Date_added __ |



    | walking | 2018-11-27T00:00:00.000 |



    | walking | 2018-11-26T00:00:00.000 |



    | running | 2018-11-24T00:00:00.000 |



    | running | 2018-11-25T00:00:00.000 |



    | walking | 2018-11-27T04:00:00.000 |




    I want to group by and count how many matches were found for the "type" field, in a certain range.
    In SQL I would do something like this:



     select type,  
    count(type)
    from index
    where date_added between '2018-11-20' and '2018-11-30'
    group by type


    I want to get something like this:




    | type | count |



    | running | 2 |



    | walking | 3 |




    I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:



    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    sourceBuilder.query(QueryBuilders
    .boolQuery()
    .must(QueryBuilders
    .rangeQuery("date_added")
    .from(start.getTime())
    .to(end.getTime()))
    )
    );


    How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?










    share|improve this question



























      0












      0








      0








      I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:




      |____Type_____|__ Date_added __ |



      | walking | 2018-11-27T00:00:00.000 |



      | walking | 2018-11-26T00:00:00.000 |



      | running | 2018-11-24T00:00:00.000 |



      | running | 2018-11-25T00:00:00.000 |



      | walking | 2018-11-27T04:00:00.000 |




      I want to group by and count how many matches were found for the "type" field, in a certain range.
      In SQL I would do something like this:



       select type,  
      count(type)
      from index
      where date_added between '2018-11-20' and '2018-11-30'
      group by type


      I want to get something like this:




      | type | count |



      | running | 2 |



      | walking | 3 |




      I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:



      SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
      sourceBuilder.query(QueryBuilders
      .boolQuery()
      .must(QueryBuilders
      .rangeQuery("date_added")
      .from(start.getTime())
      .to(end.getTime()))
      )
      );


      How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?










      share|improve this question
















      I have an instance of Elasticsearch running with thousands of documents. My index has 2 fields like this:




      |____Type_____|__ Date_added __ |



      | walking | 2018-11-27T00:00:00.000 |



      | walking | 2018-11-26T00:00:00.000 |



      | running | 2018-11-24T00:00:00.000 |



      | running | 2018-11-25T00:00:00.000 |



      | walking | 2018-11-27T04:00:00.000 |




      I want to group by and count how many matches were found for the "type" field, in a certain range.
      In SQL I would do something like this:



       select type,  
      count(type)
      from index
      where date_added between '2018-11-20' and '2018-11-30'
      group by type


      I want to get something like this:




      | type | count |



      | running | 2 |



      | walking | 3 |




      I'm using the High Level Rest Client api in my project, so far my query looks like this, it's only filtering by the start and end time:



      SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
      sourceBuilder.query(QueryBuilders
      .boolQuery()
      .must(QueryBuilders
      .rangeQuery("date_added")
      .from(start.getTime())
      .to(end.getTime()))
      )
      );


      How can I do a "group by" in the "type" field? Is it possible to do this in ElasticSearch?







      java elasticsearch






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 27 '18 at 13:37







      KimioN42

















      asked Nov 27 '18 at 13:25









      KimioN42KimioN42

      84118




      84118
























          2 Answers
          2






          active

          oldest

          votes


















          2














          That's a good start! Now you need to add a terms aggregation to your query:



          SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
          sourceBuilder.query(QueryBuilders.boolQuery()
          .must(QueryBuilders
          .rangeQuery("date_added")
          .from(start.getTime())
          .to(end.getTime()))
          )
          );

          // add these two lines
          TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
          sourceBuilder.aggregation(groupBy);





          share|improve this answer





















          • 1





            Ok, try with type.keyword instead of type as the field name. I've updated my answer

            – Val
            Nov 27 '18 at 14:36






          • 1





            Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

            – Val
            Nov 27 '18 at 14:48






          • 1





            have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

            – Val
            Nov 28 '18 at 17:22






          • 1





            Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

            – Val
            Nov 28 '18 at 18:06






          • 1





            Ok, make sure to specify which version of ES and the REST client you're running

            – Val
            Nov 28 '18 at 19:11



















          0














          After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:



          Terms  terms = searchResponse.getAggregations().get("byType");
          Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
          for (Bucket bucket : buckets) {
          System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
          }


          This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:



          Type: walking = Count(900)
          Type: running = Count(1800)





          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53500771%2felasticsearch-how-to-group-by-and-count-matches-in-an-index%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            2














            That's a good start! Now you need to add a terms aggregation to your query:



            SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
            sourceBuilder.query(QueryBuilders.boolQuery()
            .must(QueryBuilders
            .rangeQuery("date_added")
            .from(start.getTime())
            .to(end.getTime()))
            )
            );

            // add these two lines
            TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
            sourceBuilder.aggregation(groupBy);





            share|improve this answer





















            • 1





              Ok, try with type.keyword instead of type as the field name. I've updated my answer

              – Val
              Nov 27 '18 at 14:36






            • 1





              Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

              – Val
              Nov 27 '18 at 14:48






            • 1





              have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

              – Val
              Nov 28 '18 at 17:22






            • 1





              Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

              – Val
              Nov 28 '18 at 18:06






            • 1





              Ok, make sure to specify which version of ES and the REST client you're running

              – Val
              Nov 28 '18 at 19:11
















            2














            That's a good start! Now you need to add a terms aggregation to your query:



            SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
            sourceBuilder.query(QueryBuilders.boolQuery()
            .must(QueryBuilders
            .rangeQuery("date_added")
            .from(start.getTime())
            .to(end.getTime()))
            )
            );

            // add these two lines
            TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
            sourceBuilder.aggregation(groupBy);





            share|improve this answer





















            • 1





              Ok, try with type.keyword instead of type as the field name. I've updated my answer

              – Val
              Nov 27 '18 at 14:36






            • 1





              Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

              – Val
              Nov 27 '18 at 14:48






            • 1





              have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

              – Val
              Nov 28 '18 at 17:22






            • 1





              Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

              – Val
              Nov 28 '18 at 18:06






            • 1





              Ok, make sure to specify which version of ES and the REST client you're running

              – Val
              Nov 28 '18 at 19:11














            2












            2








            2







            That's a good start! Now you need to add a terms aggregation to your query:



            SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
            sourceBuilder.query(QueryBuilders.boolQuery()
            .must(QueryBuilders
            .rangeQuery("date_added")
            .from(start.getTime())
            .to(end.getTime()))
            )
            );

            // add these two lines
            TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
            sourceBuilder.aggregation(groupBy);





            share|improve this answer















            That's a good start! Now you need to add a terms aggregation to your query:



            SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
            sourceBuilder.query(QueryBuilders.boolQuery()
            .must(QueryBuilders
            .rangeQuery("date_added")
            .from(start.getTime())
            .to(end.getTime()))
            )
            );

            // add these two lines
            TermsAggregationBuilder groupBy = AggregationBuilders.terms("byType").field("type.keyword");
            sourceBuilder.aggregation(groupBy);






            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Nov 27 '18 at 14:36

























            answered Nov 27 '18 at 13:39









            ValVal

            106k6145179




            106k6145179








            • 1





              Ok, try with type.keyword instead of type as the field name. I've updated my answer

              – Val
              Nov 27 '18 at 14:36






            • 1





              Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

              – Val
              Nov 27 '18 at 14:48






            • 1





              have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

              – Val
              Nov 28 '18 at 17:22






            • 1





              Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

              – Val
              Nov 28 '18 at 18:06






            • 1





              Ok, make sure to specify which version of ES and the REST client you're running

              – Val
              Nov 28 '18 at 19:11














            • 1





              Ok, try with type.keyword instead of type as the field name. I've updated my answer

              – Val
              Nov 27 '18 at 14:36






            • 1





              Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

              – Val
              Nov 27 '18 at 14:48






            • 1





              have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

              – Val
              Nov 28 '18 at 17:22






            • 1





              Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

              – Val
              Nov 28 '18 at 18:06






            • 1





              Ok, make sure to specify which version of ES and the REST client you're running

              – Val
              Nov 28 '18 at 19:11








            1




            1





            Ok, try with type.keyword instead of type as the field name. I've updated my answer

            – Val
            Nov 27 '18 at 14:36





            Ok, try with type.keyword instead of type as the field name. I've updated my answer

            – Val
            Nov 27 '18 at 14:36




            1




            1





            Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

            – Val
            Nov 27 '18 at 14:48





            Yes, it is always best to perform aggregating and sorting on keyword fields instead of text field. type is of type text while type.keyword is of type keyword

            – Val
            Nov 27 '18 at 14:48




            1




            1





            have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

            – Val
            Nov 28 '18 at 17:22





            have you tried searchResponse.getAggregations().get("byType").getBuckets() ?

            – Val
            Nov 28 '18 at 17:22




            1




            1





            Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

            – Val
            Nov 28 '18 at 18:06





            Try this: elastic.co/guide/en/elasticsearch/client/java-rest/current/…

            – Val
            Nov 28 '18 at 18:06




            1




            1





            Ok, make sure to specify which version of ES and the REST client you're running

            – Val
            Nov 28 '18 at 19:11





            Ok, make sure to specify which version of ES and the REST client you're running

            – Val
            Nov 28 '18 at 19:11













            0














            After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:



            Terms  terms = searchResponse.getAggregations().get("byType");
            Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
            for (Bucket bucket : buckets) {
            System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
            }


            This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:



            Type: walking = Count(900)
            Type: running = Count(1800)





            share|improve this answer




























              0














              After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:



              Terms  terms = searchResponse.getAggregations().get("byType");
              Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
              for (Bucket bucket : buckets) {
              System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
              }


              This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:



              Type: walking = Count(900)
              Type: running = Count(1800)





              share|improve this answer


























                0












                0








                0







                After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:



                Terms  terms = searchResponse.getAggregations().get("byType");
                Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
                for (Bucket bucket : buckets) {
                System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
                }


                This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:



                Type: walking = Count(900)
                Type: running = Count(1800)





                share|improve this answer













                After using Val's reply to aggregate the fields, I wanted to print the aggregations of my query together with the value of them. Here's what I did:



                Terms  terms = searchResponse.getAggregations().get("byType");
                Collection<Terms.Bucket> buckets = (Collection<Bucket>) terms.getBuckets();
                for (Bucket bucket : buckets) {
                System.out.println("Type: " + bucket.getKeyAsString() + " = Count("+bucket.getDocCount()+")");
                }


                This is the output after running the query in an index with 2700 documents with a field called "type" and 2 different types:



                Type: walking = Count(900)
                Type: running = Count(1800)






                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 28 '18 at 19:36









                KimioN42KimioN42

                84118




                84118






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53500771%2felasticsearch-how-to-group-by-and-count-matches-in-an-index%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

                    Calculate evaluation metrics using cross_val_predict sklearn

                    Insert data from modal to MySQL (multiple modal on website)