Handling Zeros or NaNs in a Pandas DataFrame operations












0














I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



   08FB006 08FC001 08FC003 08FC005 08GD004

----------------------------------------------

0   253      872    256      11.80    2660

1   250      850    255      10.60    2510

2   246      850    241      10.30    2130

3   241      827    235      9.32     1970

4   241      821    229      9.17     1900

5   232       0     228      8.93     1840

6   231       0     225      8.05     1710

7   0         0     225       0       1610

8   0         0     224       0       1590

9   0         0      0        0       1590

10  0         0      0        0       1550


I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



Here is what I am trying to do with my df columns : 



shape_list1=

location_list1=

scale_list1=



for column in df.columns:

    shape1, location1, scale1=stats.genpareto.fit(df[column])



    shape_list1.append(shape1)

    location_list1.append(location1)

    scale_list1.append(scale1)





python pandas nan zero







share|improve this question





























    0














    I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



       08FB006 08FC001 08FC003 08FC005 08GD004
    
----------------------------------------------
    
0   253      872    256      11.80    2660
    
1   250      850    255      10.60    2510
    
2   246      850    241      10.30    2130
    
3   241      827    235      9.32     1970
    
4   241      821    229      9.17     1900
    
5   232       0     228      8.93     1840
    
6   231       0     225      8.05     1710
    
7   0         0     225       0       1610
    
8   0         0     224       0       1590
    
9   0         0      0        0       1590
    
10  0         0      0        0       1550


    I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



    Here is what I am trying to do with my df columns : 



    shape_list1=
    
location_list1=
    
scale_list1=
    

    
for column in df.columns:
    
    shape1, location1, scale1=stats.genpareto.fit(df[column])
    

    
    shape_list1.append(shape1)
    
    location_list1.append(location1)
    
    scale_list1.append(scale1)





    python pandas nan zero







    share|improve this question



























      0












      0








      0







      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero







      share|improve this question















      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero




      python pandas nan zero






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 22 at 22:00

























      asked Nov 22 at 21:44









      Sina Shabani

      287




      287
























          2 Answers
          2






          active

          oldest

          votes


















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41



















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41
















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer

















          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41














          1












          1








          1






          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer












          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 at 22:06









          andersource

          37416




          37416








          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41














          • 2




            Thanks both of these answers worked :)
            – Sina Shabani
            Nov 22 at 22:13












          • @andersource's syntax is a lot cleaner than mine!
            – Peter Leimbigler
            Nov 22 at 22:41








          2




          2




          Thanks both of these answers worked :)
          – Sina Shabani
          Nov 22 at 22:13






          Thanks both of these answers worked :)
          – Sina Shabani
          Nov 22 at 22:13














          @andersource's syntax is a lot cleaner than mine!
          – Peter Leimbigler
          Nov 22 at 22:41




          @andersource's syntax is a lot cleaner than mine!
          – Peter Leimbigler
          Nov 22 at 22:41













          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13
















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer





















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13














          0












          0








          0






          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer












          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 at 22:06









          Peter Leimbigler

          3,7041415




          3,7041415












          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13


















          • Thanks for the explanation :)
            – Sina Shabani
            Nov 22 at 22:13
















          Thanks for the explanation :)
          – Sina Shabani
          Nov 22 at 22:13




          Thanks for the explanation :)
          – Sina Shabani
          Nov 22 at 22:13


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Contact image not getting when fetch all contact list from iPhone by CNContact

          Image
          up vote 0 down vote favorite I know this question already asked but not getting solution. From this code I will get all the information from the contact but image not found when open vcf files on mac os, also not getting when share this file. I use this stackoverflow link here but It's not help full. var contacts = [CNContact]() let keys = [CNContactVCardSerialization.descriptorForRequiredKeys() ] as [Any] let request = CNContactFetchRequest(keysToFetch: keys as! [CNKeyDescriptor]) do { try self.contactStore.enumerateContacts(with: request) { (contact, stop) in // Array containing all unified contacts from everywhere contacts.append(contact) } } catch { print("unable to fetch contacts") } do { let data = try
          Read more

          count number of partitions of a set with n elements into k subsets

          Image
          up vote 6 down vote favorite 1 This program is for count number of partitions of a set with n elements into k subsets I am confusing here return k*countP(n-1, k) + countP(n-1, k-1); can some one explain what is happening here? why we are multiplying with k? NOTE->I know this is not the best way to calculate number of partitions that would be DP // A C++ program to count number of partitions // of a set with n elements into k subsets #include<iostream> using namespace std; // Returns count of different partitions of n // elements in k subsets int countP(int n, int k) { // Base cases if (n == 0 || k == 0 || k > n) return 0; if (k == 1 || k == n) return 1; // S(n+1, k) = k*S(n, k) + S(n, k-1) return k*countP(n-1, k) + countP(n-1, k-1); } // Driver program int ma
          Read more

          A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

          Image
          0 I found a lot of questions abount appendices and ToC. Many users want appendices to be grouped in an Appendix part, however some problems arise with ToC, hyperref, PDF viewer bookmarks, and so on. There are different solutions which require extra packages, command patching and other extra code, however none of them satisfies me. I almost found an easy way to accomplish a good result, where appendices are added to bookmarks in the right way and hyperref links point to the right page. However, the number of the "Appendix" part page is wrong (it's the number of appendix A). Is there any EASY way to fix that? This is a MWE: documentclass{book} usepackage[nottoc,notlot,notlof]{tocbibind} usepackage{hyperref} begin{document} frontmatter tableofcontents mainmatter part{First} chapter{
          Read more