Btukfyl

I have a pandas dataframe, loaded from a CSV, with one column that has encoded unicode characters like u00ca. the str.normalize() method should take care of these, but it's not working, even when it works with unicodedata.normalize

import unicodedata

s = 'BC - CPE LE Hu00caTRE INC.'

unicodedata.normalize('NFKD', s)

>> 'BC - CPE LE HÊTRE INC.'

But not when it's in a pandas series.

import pandas as pd



names = ['BC - CPE LE Hu00caTRE INC.',

 'BC - CPE LE CHEZ-MOI DES PETITS',

 'BC GARDE MILIEU FAMILIAL DE BORDEAUX-CARTIERVILLE',

 'BC - BCGMF AHUNSTIC',

 'BC - CPE LE JARDIN DES Ru00caVES INC.',

 'BC - FORCE VIVE" CPE"',

 'BC - CPE GAMINVILLE INC.',

 'BC - CPE PIROUETTE DE FABREVILLE INC.',

 'B.C. ST-MICHEL',

 'BC - CPE DU PARC',

 'BC - CPE LA TROTTINETTE CAROTTEE',

 'BC - CPE DE MONTRu00c9AL-NORD']



names = pd.Series(names)

names.str.normalize('NFKD')



>> 0                           BC - CPE LE Hu00caTRE INC.

  1                       BC - CPE LE CHEZ-MOI DES PETITS

  2     BC GARDE MILIEU FAMILIAL DE BORDEAUX-CARTIERVILLE

  3                                   BC - BCGMF AHUNSTIC

  4                BC - CPE LE JARDIN DES Ru00caVES INC.

  5                               BC - FORCE VIVE" CPE"

  6                              BC - CPE GAMINVILLE INC.

  7                 BC - CPE PIROUETTE DE FABREVILLE INC.

  8                                        B.C. ST-MICHEL

  9                                      BC - CPE DU PARC

  10                     BC - CPE LA TROTTINETTE CAROTTEE

  11                       BC - CPE DE MONTRu00c9AL-NORD

  dtype: object

I have also tried every variation possible of str.encode and str.decode before and after normalize. Nothing changed.

unicodedata.normalize isn't doing what you think it is. unicodedata.normalize does not process u escape sequences; it converts input into various Unicode normalization forms.

Python string literal processing is what converts the u00ca to an Ê character, and Python string literal processing is not applied to anything but Python string literals. The input you're reading from a CSV file does not get Python string literal processing applied. (The contents of the names list in your question do get string literal processing applied, so your posted code fails to reproduce your error. You really should have checked that before posting.)

Depending on the content of the file and the context of your application, decoding your input with the unicode-escape encoding using codecs.decode may be an appropriate way to handle the u escapes.

The problem was with pandas escaping the unicode character in the string. So u00ca was being saved as \u00ca. To decode it back, I just needed this, which @user2357112 hinted at:

Series.str.decode('unicode-escape')