Btukfyl

This question is already resolved, but...

...also consider the solution suggested by Wouter in his original comment. The ability to handle missing data, including dropna(), is built into pandas explicitly. Aside from potentially improved performance over doing it manually, these functions also come with a variety of options which may be useful.

In [24]: df = pd.DataFrame(np.random.randn(10,3))



In [25]: df.iloc[::2,0] = np.nan; df.iloc[::4,1] = np.nan; df.iloc[::3,2] = np.nan;



In [26]: df

Out[26]:

          0         1         2

0       NaN       NaN       NaN

1  2.677677 -1.466923 -0.750366

2       NaN  0.798002 -0.906038

3  0.672201  0.964789       NaN

4       NaN       NaN  0.050742

5 -1.250970  0.030561 -2.678622

6       NaN  1.036043       NaN

7  0.049896 -0.308003  0.823295

8       NaN       NaN  0.637482

9 -0.310130  0.078891       NaN

In [27]: df.dropna()     #drop all rows that have any NaN values

Out[27]:

          0         1         2

1  2.677677 -1.466923 -0.750366

5 -1.250970  0.030561 -2.678622

7  0.049896 -0.308003  0.823295

In [28]: df.dropna(how='all')     #drop only if ALL columns are NaN

Out[28]:

          0         1         2

1  2.677677 -1.466923 -0.750366

2       NaN  0.798002 -0.906038

3  0.672201  0.964789       NaN

4       NaN       NaN  0.050742

5 -1.250970  0.030561 -2.678622

6       NaN  1.036043       NaN

7  0.049896 -0.308003  0.823295

8       NaN       NaN  0.637482

9 -0.310130  0.078891       NaN

In [29]: df.dropna(thresh=2)   #Drop row if it does not have at least two values that are **not** NaN

Out[29]:

          0         1         2

1  2.677677 -1.466923 -0.750366

2       NaN  0.798002 -0.906038

3  0.672201  0.964789       NaN

5 -1.250970  0.030561 -2.678622

7  0.049896 -0.308003  0.823295

9 -0.310130  0.078891       NaN

In [30]: df.dropna(subset=[1])   #Drop only if NaN in specific column (as asked in the question)

Out[30]:

          0         1         2

1  2.677677 -1.466923 -0.750366

2       NaN  0.798002 -0.906038

3  0.672201  0.964789       NaN

5 -1.250970  0.030561 -2.678622

6       NaN  1.036043       NaN

7  0.049896 -0.308003  0.823295

9 -0.310130  0.078891       NaN

There are also other options (See docs at http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html), including dropping columns instead of rows.

Pretty handy!

I know this has already been answered, but just for the sake of a purely pandas solution to this specific question as opposed to the general description from Aman (which was wonderful) and in case anyone else happens upon this:

import pandas as pd

df = df[pd.notnull(df['EPS'])]

You can use this:

df.dropna(subset=['EPS'], how='all', inplace = True)

Simplest of all solutions:

filtered_df = df[df['EPS'].notnull()]

The above solution is way better than using np.isfinite()

You could use dataframe method notnull or inverse of isnull, or numpy.isnan:

In [332]: df[df.EPS.notnull()]

Out[332]:

   STK_ID  RPT_Date  STK_ID.1  EPS  cash

2  600016  20111231    600016  4.3   NaN

4  601939  20111231    601939  2.5   NaN





In [334]: df[~df.EPS.isnull()]

Out[334]:

   STK_ID  RPT_Date  STK_ID.1  EPS  cash

2  600016  20111231    600016  4.3   NaN

4  601939  20111231    601939  2.5   NaN





In [347]: df[~np.isnan(df.EPS)]

Out[347]:

   STK_ID  RPT_Date  STK_ID.1  EPS  cash

2  600016  20111231    600016  4.3   NaN

4  601939  20111231    601939  2.5   NaN

yet another solution which uses the fact that np.nan != np.nan:

In [149]: df.query("EPS == EPS")

Out[149]:

                 STK_ID  EPS  cash

STK_ID RPT_Date

600016 20111231  600016  4.3   NaN

601939 20111231  601939  2.5   NaN

you can use dropna

Example

Drop the rows where at least one element is missing.

df=df.dropna()

Define in which columns to look for missing values.

df=df.dropna(subset=['column1', 'column1'])

See this for more examples

Note: axis parameter of dropna is deprecated since version 0.23.0:

Or (check for NaN's with isnull, then use ~ to make the opposite to no NaN's):

df=df[~df['EPS'].isnull()]

Now:

print(df)

Is:

                 STK_ID  EPS  cash

STK_ID RPT_Date

600016 20111231  600016  4.3   NaN

601939 20111231  601939  2.5   NaN

It may be added at that '&' can be used to add additional conditions e.g.

df = df[(df.EPS > 2.0) & (df.EPS <4.0)]

Notice that when evaluating the statements, pandas needs parenthesis.

Simple and easy way

df.dropna(subset=['EPS'],inplace=True)

source: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html

For some reason none of the previously submitted answers worked for me. This basic solution did:

df = df[df.EPS >= 0]

Though of course that will drop rows with negative numbers, too. So if you want those it's probably smart to add this after, too.

df = df[df.EPS <= 0]