Handling Zeros or NaNs in a Pandas DataFrame operations












0














I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



   08FB006 08FC001 08FC003 08FC005 08GD004

----------------------------------------------

0   253      872    256      11.80    2660

1   250      850    255      10.60    2510

2   246      850    241      10.30    2130

3   241      827    235      9.32     1970

4   241      821    229      9.17     1900

5   232       0     228      8.93     1840

6   231       0     225      8.05     1710

7   0         0     225       0       1610

8   0         0     224       0       1590

9   0         0      0        0       1590

10  0         0      0        0       1550


I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



Here is what I am trying to do with my df columns : 



shape_list1=

location_list1=

scale_list1=



for column in df.columns:

    shape1, location1, scale1=stats.genpareto.fit(df[column])



    shape_list1.append(shape1)

    location_list1.append(location1)

    scale_list1.append(scale1)





python pandas nan zero







share|improve this question





























    0














    I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



       08FB006 08FC001 08FC003 08FC005 08GD004
    
----------------------------------------------
    
0   253      872    256      11.80    2660
    
1   250      850    255      10.60    2510
    
2   246      850    241      10.30    2130
    
3   241      827    235      9.32     1970
    
4   241      821    229      9.17     1900
    
5   232       0     228      8.93     1840
    
6   231       0     225      8.05     1710
    
7   0         0     225       0       1610
    
8   0         0     224       0       1590
    
9   0         0      0        0       1590
    
10  0         0      0        0       1550


    I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



    Here is what I am trying to do with my df columns : 



    shape_list1=
    
location_list1=
    
scale_list1=
    

    
for column in df.columns:
    
    shape1, location1, scale1=stats.genpareto.fit(df[column])
    

    
    shape_list1.append(shape1)
    
    location_list1.append(location1)
    
    scale_list1.append(scale1)





    python pandas nan zero







    share|improve this question



























      0












      0








      0







      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero







      share|improve this question















      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero




      python pandas nan zero






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 22 at 22:00

























      asked Nov 22 at 21:44









      Sina Shabani

      287




      287
























          2 Answers
          2






          active

          oldest

          votes


















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41



















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41
















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41














          1












          1








          1






          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer












          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 at 22:06









          andersource

          37416




          37416








          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41














          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41








          2




          2




          Thanks both of these answers worked :)
          – Sina Shabani
          Nov 22 at 22:13






          Thanks both of these answers worked :)
          – Sina Shabani
          Nov 22 at 22:13














          @andersource's syntax is a lot cleaner than mine!
          – Peter Leimbigler
          Nov 22 at 22:41




          @andersource's syntax is a lot cleaner than mine!
          – Peter Leimbigler
          Nov 22 at 22:41













          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13
















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13














          0












          0








          0






          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer












          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 at 22:06









          Peter Leimbigler

          3,7041415




          3,7041415












          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13


















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13
















          Thanks for the explanation :)
          – Sina Shabani
          Nov 22 at 22:13




          Thanks for the explanation :)
          – Sina Shabani
          Nov 22 at 22:13


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

          Image
          0 I found a lot of questions abount appendices and ToC. Many users want appendices to be grouped in an Appendix part, however some problems arise with ToC, hyperref, PDF viewer bookmarks, and so on. There are different solutions which require extra packages, command patching and other extra code, however none of them satisfies me. I almost found an easy way to accomplish a good result, where appendices are added to bookmarks in the right way and hyperref links point to the right page. However, the number of the "Appendix" part page is wrong (it's the number of appendix A). Is there any EASY way to fix that? This is a MWE: documentclass{book} usepackage[nottoc,notlot,notlof]{tocbibind} usepackage{hyperref} begin{document} frontmatter tableofcontents mainmatter part{First} chapter{...
          Read more

          Calculate evaluation metrics using cross_val_predict sklearn

          Image
          1 In the sklearn.model_selection.cross_val_predict page it is stated: Generate cross-validated estimates for each input data point. It is not appropriate to pass these predictions into an evaluation metric. Can someone explain what does it mean? If this gives estimate of Y (y prediction) for every Y (true Y), why can't I calculate metrics such as RMSE or coefficient of determination using these results? python scikit-learn cross-validation share | improve this question edited Nov 28 '18 at 17:52 desertnaut 20.3k 7 43 79 ...
          Read more

          Insert data from modal to MySQL (multiple modal on website)

          Image
          0 I have problem with inserting data to MySQL from modal. My modal: <a href="#" class="badge badge-pill badge-success">6 komentarzy</a> <a href="#" class="badge badge-pill badge-danger">brak komentarzy</a> <a data-toggle="modal" href="#add_desk_comm_{$desk_['desk_id']}" data-target="#add_desk_comm_{$desk_['desk_id']}" class="ediiit">(dodaj)</a> <div class="modal fade" id="add_desk_comm_{$desk_['desk_id']}" tabindex="-1" role="dialog" aria-labelledby="edit_printer" aria-hidden="true"> <div class="modal-dialog" role="document"> ...
          Read more