Spark - combine filter results from all executors












4














I have 3 executors in my spark streaming job which consumes from Kafka. Executor count depends on partition count in topic. When a message consumed from this topic, I am starting query on Hazelcast. Every executor finds results from some filtering operation on hazelcast and returns duplicated results. Because data statuses are not updated when executor returns the data and other executor finds the same data.



My question is, is there a way to combine all results in only one list which are found by executors during streaming?






java apache-spark hazelcast







share|improve this question
























  • use accumulators...pls share ur code..
    – Taha Naqvi
    Nov 23 at 10:23












  • thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
    – Masay
    Nov 23 at 11:22
















4














I have 3 executors in my spark streaming job which consumes from Kafka. Executor count depends on partition count in topic. When a message consumed from this topic, I am starting query on Hazelcast. Every executor finds results from some filtering operation on hazelcast and returns duplicated results. Because data statuses are not updated when executor returns the data and other executor finds the same data.



My question is, is there a way to combine all results in only one list which are found by executors during streaming?






java apache-spark hazelcast







share|improve this question
























  • use accumulators...pls share ur code..
    – Taha Naqvi
    Nov 23 at 10:23












  • thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
    – Masay
    Nov 23 at 11:22














4












4








4







I have 3 executors in my spark streaming job which consumes from Kafka. Executor count depends on partition count in topic. When a message consumed from this topic, I am starting query on Hazelcast. Every executor finds results from some filtering operation on hazelcast and returns duplicated results. Because data statuses are not updated when executor returns the data and other executor finds the same data.



My question is, is there a way to combine all results in only one list which are found by executors during streaming?






java apache-spark hazelcast







share|improve this question















I have 3 executors in my spark streaming job which consumes from Kafka. Executor count depends on partition count in topic. When a message consumed from this topic, I am starting query on Hazelcast. Every executor finds results from some filtering operation on hazelcast and returns duplicated results. Because data statuses are not updated when executor returns the data and other executor finds the same data.



My question is, is there a way to combine all results in only one list which are found by executors during streaming?






java apache-spark hazelcast




java apache-spark hazelcast






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 23 at 11:21

























asked Nov 23 at 9:00









Masay

55121131




55121131












  • use accumulators...pls share ur code..
    – Taha Naqvi
    Nov 23 at 10:23












  • thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
    – Masay
    Nov 23 at 11:22


















  • use accumulators...pls share ur code..
    – Taha Naqvi
    Nov 23 at 10:23












  • thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
    – Masay
    Nov 23 at 11:22
















use accumulators...pls share ur code..
– Taha Naqvi
Nov 23 at 10:23






use accumulators...pls share ur code..
– Taha Naqvi
Nov 23 at 10:23














thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
– Masay
Nov 23 at 11:22




thx for your comment. I detailed my question. Accumulator is still on the table and I am reading about it.
– Masay
Nov 23 at 11:22












2 Answers
2






active

oldest

votes


















0














Spark Executors are distributed across Cluster, so if you are trying to deduplicate data across cluster. So deduplicating is difficult. you have following options




  1. Use accumulators.- problem here is that accumulators are not consistent when job is running and you may end up reading stale data

  2. Other option is Offload this work to external system. - store your output in some external storage which can deduplicate it. (Probably HBase). efficiency of this storage system becomes key here.


I hope this helps






share|improve this answer





























    0














    To avoid duplicate data read, you need to maintain the offset somewhere, preferred in HBase and everytime you consume the data from Kafka, you read it from HBase and then check the offset for each topic which is already consumed and then start reading and writing it. After each successful write, you must update the offset count.



    Do you think that way it solves the issue?






    share|improve this answer





















      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53443456%2fspark-combine-filter-results-from-all-executors%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      0














      Spark Executors are distributed across Cluster, so if you are trying to deduplicate data across cluster. So deduplicating is difficult. you have following options




      1. Use accumulators.- problem here is that accumulators are not consistent when job is running and you may end up reading stale data

      2. Other option is Offload this work to external system. - store your output in some external storage which can deduplicate it. (Probably HBase). efficiency of this storage system becomes key here.


      I hope this helps






      share|improve this answer


























        0














        Spark Executors are distributed across Cluster, so if you are trying to deduplicate data across cluster. So deduplicating is difficult. you have following options




        1. Use accumulators.- problem here is that accumulators are not consistent when job is running and you may end up reading stale data

        2. Other option is Offload this work to external system. - store your output in some external storage which can deduplicate it. (Probably HBase). efficiency of this storage system becomes key here.


        I hope this helps






        share|improve this answer
























          0












          0








          0






          Spark Executors are distributed across Cluster, so if you are trying to deduplicate data across cluster. So deduplicating is difficult. you have following options




          1. Use accumulators.- problem here is that accumulators are not consistent when job is running and you may end up reading stale data

          2. Other option is Offload this work to external system. - store your output in some external storage which can deduplicate it. (Probably HBase). efficiency of this storage system becomes key here.


          I hope this helps






          share|improve this answer












          Spark Executors are distributed across Cluster, so if you are trying to deduplicate data across cluster. So deduplicating is difficult. you have following options




          1. Use accumulators.- problem here is that accumulators are not consistent when job is running and you may end up reading stale data

          2. Other option is Offload this work to external system. - store your output in some external storage which can deduplicate it. (Probably HBase). efficiency of this storage system becomes key here.


          I hope this helps







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 28 at 6:47









          Harjeet Kumar

          1562




          1562

























              0














              To avoid duplicate data read, you need to maintain the offset somewhere, preferred in HBase and everytime you consume the data from Kafka, you read it from HBase and then check the offset for each topic which is already consumed and then start reading and writing it. After each successful write, you must update the offset count.



              Do you think that way it solves the issue?






              share|improve this answer


























                0














                To avoid duplicate data read, you need to maintain the offset somewhere, preferred in HBase and everytime you consume the data from Kafka, you read it from HBase and then check the offset for each topic which is already consumed and then start reading and writing it. After each successful write, you must update the offset count.



                Do you think that way it solves the issue?






                share|improve this answer
























                  0












                  0








                  0






                  To avoid duplicate data read, you need to maintain the offset somewhere, preferred in HBase and everytime you consume the data from Kafka, you read it from HBase and then check the offset for each topic which is already consumed and then start reading and writing it. After each successful write, you must update the offset count.



                  Do you think that way it solves the issue?






                  share|improve this answer












                  To avoid duplicate data read, you need to maintain the offset somewhere, preferred in HBase and everytime you consume the data from Kafka, you read it from HBase and then check the offset for each topic which is already consumed and then start reading and writing it. After each successful write, you must update the offset count.



                  Do you think that way it solves the issue?







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Nov 29 at 14:24









                  H Roy

                  11416




                  11416






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.





                      Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                      Please pay close attention to the following guidance:


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53443456%2fspark-combine-filter-results-from-all-executors%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      Contact image not getting when fetch all contact list from iPhone by CNContact

                      Image
                      up vote 0 down vote favorite I know this question already asked but not getting solution. From this code I will get all the information from the contact but image not found when open vcf files on mac os, also not getting when share this file. I use this stackoverflow link here but It's not help full. var contacts = [CNContact]() let keys = [CNContactVCardSerialization.descriptorForRequiredKeys() ] as [Any] let request = CNContactFetchRequest(keysToFetch: keys as! [CNKeyDescriptor]) do { try self.contactStore.enumerateContacts(with: request) { (contact, stop) in // Array containing all unified contacts from everywhere contacts.append(contact) } } catch { print("unable to fetch contacts") } do { let data = try
                      Read more

                      count number of partitions of a set with n elements into k subsets

                      Image
                      up vote 6 down vote favorite 1 This program is for count number of partitions of a set with n elements into k subsets I am confusing here return k*countP(n-1, k) + countP(n-1, k-1); can some one explain what is happening here? why we are multiplying with k? NOTE->I know this is not the best way to calculate number of partitions that would be DP // A C++ program to count number of partitions // of a set with n elements into k subsets #include<iostream> using namespace std; // Returns count of different partitions of n // elements in k subsets int countP(int n, int k) { // Base cases if (n == 0 || k == 0 || k > n) return 0; if (k == 1 || k == n) return 1; // S(n+1, k) = k*S(n, k) + S(n, k-1) return k*countP(n-1, k) + countP(n-1, k-1); } // Driver program int ma
                      Read more

                      A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

                      Image
                      0 I found a lot of questions abount appendices and ToC. Many users want appendices to be grouped in an Appendix part, however some problems arise with ToC, hyperref, PDF viewer bookmarks, and so on. There are different solutions which require extra packages, command patching and other extra code, however none of them satisfies me. I almost found an easy way to accomplish a good result, where appendices are added to bookmarks in the right way and hyperref links point to the right page. However, the number of the "Appendix" part page is wrong (it's the number of appendix A). Is there any EASY way to fix that? This is a MWE: documentclass{book} usepackage[nottoc,notlot,notlof]{tocbibind} usepackage{hyperref} begin{document} frontmatter tableofcontents mainmatter part{First} chapter{
                      Read more