Undesired latency when kernel thread does more communication work












1















We have developed a kernel module that acts basically like a wired communication traffic (eth, ...) - WiFi bridge. It forwards periodically incoming wired data to WiFi and vice versa. The system consists of two devices running the same kernel module. Under certain conditions we are experiencing undesired latency.



The module starts a kernel thread which basically performs:



while(1) {
schedule()
do_bridge_work()
usleep_range(200, 500)
}


During normal operation, top command displays:



CPU:   0% usr  19% sys   0% nic  60% idle   0% io   0% irq  19% sirq 
Load average: 1.03 1.06 1.01 1/54 491
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
491 2 root DW 0 0% 29% [bridge_thread]
3 2 root SW 0 0% 10% [ksoftirqd/0]


When diagnostics is activated the thread also sends out additional diagnostic wired eth packets. In this mode, ping of devices on one side to the other side goes from 5-6ms to 45-900 ms. top then shows:



CPU:   0% usr  25% sys   0% nic  50% idle   0% io   0% irq  24% sirq 
Load average: 1.06 1.07 1.01 1/54 491
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
491 2 root DW 0 0% 35% [bridge_thread]
3 2 root SW 0 0% 14% [ksoftirqd/0]


If an additional schedule() is inserted before usleep_range(), or if the sleep time is increased, it drastically reduces the latency. I think I have narrowed it down to conclude that the RX softirq does not get the scheduling time it needs to process the incoming traffic (NET_RX_SOFTIRQ). This is based on the fact the timing of transmitted packets are perfectly fine.



My questions are:



1: Why does the ping time increase due to more work in do_bridge_work() (more processing and additional transmitting of packets). Is NET_RX_SOFTIRQ likely starved?



2: Why does the ping time decrease when additional schedule() is inserted, or when sleep time is increased? Does that mean that 200-500 us is not enough to process all pending softirqs work, and the lower softirqs are starved? Is the effect of adding additional schedule() the same as letting other do more work by increasing the sleep time?



Kernel version is 4.1.38, configured with NO_HZ, PREEMPT = y.










share|improve this question





























    1















    We have developed a kernel module that acts basically like a wired communication traffic (eth, ...) - WiFi bridge. It forwards periodically incoming wired data to WiFi and vice versa. The system consists of two devices running the same kernel module. Under certain conditions we are experiencing undesired latency.



    The module starts a kernel thread which basically performs:



    while(1) {
    schedule()
    do_bridge_work()
    usleep_range(200, 500)
    }


    During normal operation, top command displays:



    CPU:   0% usr  19% sys   0% nic  60% idle   0% io   0% irq  19% sirq 
    Load average: 1.03 1.06 1.01 1/54 491
    PID PPID USER STAT VSZ %VSZ %CPU COMMAND
    491 2 root DW 0 0% 29% [bridge_thread]
    3 2 root SW 0 0% 10% [ksoftirqd/0]


    When diagnostics is activated the thread also sends out additional diagnostic wired eth packets. In this mode, ping of devices on one side to the other side goes from 5-6ms to 45-900 ms. top then shows:



    CPU:   0% usr  25% sys   0% nic  50% idle   0% io   0% irq  24% sirq 
    Load average: 1.06 1.07 1.01 1/54 491
    PID PPID USER STAT VSZ %VSZ %CPU COMMAND
    491 2 root DW 0 0% 35% [bridge_thread]
    3 2 root SW 0 0% 14% [ksoftirqd/0]


    If an additional schedule() is inserted before usleep_range(), or if the sleep time is increased, it drastically reduces the latency. I think I have narrowed it down to conclude that the RX softirq does not get the scheduling time it needs to process the incoming traffic (NET_RX_SOFTIRQ). This is based on the fact the timing of transmitted packets are perfectly fine.



    My questions are:



    1: Why does the ping time increase due to more work in do_bridge_work() (more processing and additional transmitting of packets). Is NET_RX_SOFTIRQ likely starved?



    2: Why does the ping time decrease when additional schedule() is inserted, or when sleep time is increased? Does that mean that 200-500 us is not enough to process all pending softirqs work, and the lower softirqs are starved? Is the effect of adding additional schedule() the same as letting other do more work by increasing the sleep time?



    Kernel version is 4.1.38, configured with NO_HZ, PREEMPT = y.










    share|improve this question



























      1












      1








      1








      We have developed a kernel module that acts basically like a wired communication traffic (eth, ...) - WiFi bridge. It forwards periodically incoming wired data to WiFi and vice versa. The system consists of two devices running the same kernel module. Under certain conditions we are experiencing undesired latency.



      The module starts a kernel thread which basically performs:



      while(1) {
      schedule()
      do_bridge_work()
      usleep_range(200, 500)
      }


      During normal operation, top command displays:



      CPU:   0% usr  19% sys   0% nic  60% idle   0% io   0% irq  19% sirq 
      Load average: 1.03 1.06 1.01 1/54 491
      PID PPID USER STAT VSZ %VSZ %CPU COMMAND
      491 2 root DW 0 0% 29% [bridge_thread]
      3 2 root SW 0 0% 10% [ksoftirqd/0]


      When diagnostics is activated the thread also sends out additional diagnostic wired eth packets. In this mode, ping of devices on one side to the other side goes from 5-6ms to 45-900 ms. top then shows:



      CPU:   0% usr  25% sys   0% nic  50% idle   0% io   0% irq  24% sirq 
      Load average: 1.06 1.07 1.01 1/54 491
      PID PPID USER STAT VSZ %VSZ %CPU COMMAND
      491 2 root DW 0 0% 35% [bridge_thread]
      3 2 root SW 0 0% 14% [ksoftirqd/0]


      If an additional schedule() is inserted before usleep_range(), or if the sleep time is increased, it drastically reduces the latency. I think I have narrowed it down to conclude that the RX softirq does not get the scheduling time it needs to process the incoming traffic (NET_RX_SOFTIRQ). This is based on the fact the timing of transmitted packets are perfectly fine.



      My questions are:



      1: Why does the ping time increase due to more work in do_bridge_work() (more processing and additional transmitting of packets). Is NET_RX_SOFTIRQ likely starved?



      2: Why does the ping time decrease when additional schedule() is inserted, or when sleep time is increased? Does that mean that 200-500 us is not enough to process all pending softirqs work, and the lower softirqs are starved? Is the effect of adding additional schedule() the same as letting other do more work by increasing the sleep time?



      Kernel version is 4.1.38, configured with NO_HZ, PREEMPT = y.










      share|improve this question
















      We have developed a kernel module that acts basically like a wired communication traffic (eth, ...) - WiFi bridge. It forwards periodically incoming wired data to WiFi and vice versa. The system consists of two devices running the same kernel module. Under certain conditions we are experiencing undesired latency.



      The module starts a kernel thread which basically performs:



      while(1) {
      schedule()
      do_bridge_work()
      usleep_range(200, 500)
      }


      During normal operation, top command displays:



      CPU:   0% usr  19% sys   0% nic  60% idle   0% io   0% irq  19% sirq 
      Load average: 1.03 1.06 1.01 1/54 491
      PID PPID USER STAT VSZ %VSZ %CPU COMMAND
      491 2 root DW 0 0% 29% [bridge_thread]
      3 2 root SW 0 0% 10% [ksoftirqd/0]


      When diagnostics is activated the thread also sends out additional diagnostic wired eth packets. In this mode, ping of devices on one side to the other side goes from 5-6ms to 45-900 ms. top then shows:



      CPU:   0% usr  25% sys   0% nic  50% idle   0% io   0% irq  24% sirq 
      Load average: 1.06 1.07 1.01 1/54 491
      PID PPID USER STAT VSZ %VSZ %CPU COMMAND
      491 2 root DW 0 0% 35% [bridge_thread]
      3 2 root SW 0 0% 14% [ksoftirqd/0]


      If an additional schedule() is inserted before usleep_range(), or if the sleep time is increased, it drastically reduces the latency. I think I have narrowed it down to conclude that the RX softirq does not get the scheduling time it needs to process the incoming traffic (NET_RX_SOFTIRQ). This is based on the fact the timing of transmitted packets are perfectly fine.



      My questions are:



      1: Why does the ping time increase due to more work in do_bridge_work() (more processing and additional transmitting of packets). Is NET_RX_SOFTIRQ likely starved?



      2: Why does the ping time decrease when additional schedule() is inserted, or when sleep time is increased? Does that mean that 200-500 us is not enough to process all pending softirqs work, and the lower softirqs are starved? Is the effect of adding additional schedule() the same as letting other do more work by increasing the sleep time?



      Kernel version is 4.1.38, configured with NO_HZ, PREEMPT = y.







      c linux-kernel scheduling preemption






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 28 '18 at 14:02









      red0ct

      1,32931023




      1,32931023










      asked Nov 28 '18 at 7:17









      user2818825user2818825

      233




      233
























          0






          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53514056%2fundesired-latency-when-kernel-thread-does-more-communication-work%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53514056%2fundesired-latency-when-kernel-thread-does-more-communication-work%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

          Calculate evaluation metrics using cross_val_predict sklearn

          Insert data from modal to MySQL (multiple modal on website)