Collecting Duplicate Column names from multiple DataFrames Python
1
Essentially I am trying to create a list so that I can merge my Dataframes on Column names that are duplicated. Below is how I am currently set up.
sheets = [df1, df2, df3, df4]
col_list =
dup_list =
for i in sheets:
col_list.append(i.columns.values)
for c in col_list:
if c.duplicated():
dup_list.append(c)
I get the following error
AttributeError: 'numpy.ndarray' object has no attribute 'duplicated'
I know there is no method for duplicated. What is the best way to get the duplicated column names that appear in all dataframes.
Any help is greatly appreciated.
python pandas merge data-modeling
asked Nov 25 '18 at 15:22
PaulPaul
305
add a comment |
1
Essentially I am trying to create a list so that I can merge my Dataframes on Column names that are duplicated. Below is how I am currently set up.
sheets = [df1, df2, df3, df4]
col_list =
dup_list =
for i in sheets:
col_list.append(i.columns.values)
for c in col_list:
if c.duplicated():
dup_list.append(c)
I get the following error
AttributeError: 'numpy.ndarray' object has no attribute 'duplicated'
I know there is no method for duplicated. What is the best way to get the duplicated column names that appear in all dataframes.
Any help is greatly appreciated.
python pandas merge data-modeling
asked Nov 25 '18 at 15:22
PaulPaul
305
add a comment |
1
1
1
Essentially I am trying to create a list so that I can merge my Dataframes on Column names that are duplicated. Below is how I am currently set up.
sheets = [df1, df2, df3, df4]
col_list =
dup_list =
for i in sheets:
col_list.append(i.columns.values)
for c in col_list:
if c.duplicated():
dup_list.append(c)
I get the following error
AttributeError: 'numpy.ndarray' object has no attribute 'duplicated'
I know there is no method for duplicated. What is the best way to get the duplicated column names that appear in all dataframes.
Any help is greatly appreciated.
python pandas merge data-modeling
asked Nov 25 '18 at 15:22
PaulPaul
305
Essentially I am trying to create a list so that I can merge my Dataframes on Column names that are duplicated. Below is how I am currently set up.
sheets = [df1, df2, df3, df4]
col_list =
dup_list =
for i in sheets:
col_list.append(i.columns.values)
for c in col_list:
if c.duplicated():
dup_list.append(c)
I get the following error
AttributeError: 'numpy.ndarray' object has no attribute 'duplicated'
I know there is no method for duplicated. What is the best way to get the duplicated column names that appear in all dataframes.
Any help is greatly appreciated.
python pandas merge data-modeling
python pandas merge data-modeling
asked Nov 25 '18 at 15:22
PaulPaul
305
asked Nov 25 '18 at 15:22
PaulPaul
305
asked Nov 25 '18 at 15:22
PaulPaul
305
asked Nov 25 '18 at 15:22
PaulPaul
305
asked Nov 25 '18 at 15:22
PaulPaul
305
305
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
2
I believe you need set.intersection with mapped all columns names to sets:
df1 = pd.DataFrame(columns=list('acbd'))
df2 = pd.DataFrame(columns=list('abde'))
df3 = pd.DataFrame(columns=list('cbad'))
df4 = pd.DataFrame(columns=list('acbf'))
sheets = [df1, df2, df3, df4]
L = [x.columns for x in sheets]
#less readable
#L = [x for x in sheets]
dup_list = list(set.intersection(*map(set,L)))
print (dup_list)
['a', 'b']
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in allsheetnames:( Or maybe some whitespaces? Is possible seeL = [x.columns for x in sheets]print (L)?
– jezrael
Nov 25 '18 at 15:42
1
So if testL = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets]what isprint (L)?
– jezrael
Nov 25 '18 at 16:04
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually bydf4.columns = ['cola1','col2',...]. And then better is useconcatlikemerge- Usedfs = [x.set_index(dup_list) for x in sheets]withdf = pd.concat(dfs, axis=1).reset_index()
– jezrael
Nov 25 '18 at 16:16
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
|
show 5 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53468943%2fcollecting-duplicate-column-names-from-multiple-dataframes-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
2
I believe you need set.intersection with mapped all columns names to sets:
df1 = pd.DataFrame(columns=list('acbd'))
df2 = pd.DataFrame(columns=list('abde'))
df3 = pd.DataFrame(columns=list('cbad'))
df4 = pd.DataFrame(columns=list('acbf'))
sheets = [df1, df2, df3, df4]
L = [x.columns for x in sheets]
#less readable
#L = [x for x in sheets]
dup_list = list(set.intersection(*map(set,L)))
print (dup_list)
['a', 'b']
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in allsheetnames:( Or maybe some whitespaces? Is possible seeL = [x.columns for x in sheets]print (L)?
– jezrael
Nov 25 '18 at 15:42
1
So if testL = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets]what isprint (L)?
– jezrael
Nov 25 '18 at 16:04
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually bydf4.columns = ['cola1','col2',...]. And then better is useconcatlikemerge- Usedfs = [x.set_index(dup_list) for x in sheets]withdf = pd.concat(dfs, axis=1).reset_index()
– jezrael
Nov 25 '18 at 16:16
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
|
show 5 more comments
2
I believe you need set.intersection with mapped all columns names to sets:
df1 = pd.DataFrame(columns=list('acbd'))
df2 = pd.DataFrame(columns=list('abde'))
df3 = pd.DataFrame(columns=list('cbad'))
df4 = pd.DataFrame(columns=list('acbf'))
sheets = [df1, df2, df3, df4]
L = [x.columns for x in sheets]
#less readable
#L = [x for x in sheets]
dup_list = list(set.intersection(*map(set,L)))
print (dup_list)
['a', 'b']
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in allsheetnames:( Or maybe some whitespaces? Is possible seeL = [x.columns for x in sheets]print (L)?
– jezrael
Nov 25 '18 at 15:42
1
So if testL = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets]what isprint (L)?
– jezrael
Nov 25 '18 at 16:04
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually bydf4.columns = ['cola1','col2',...]. And then better is useconcatlikemerge- Usedfs = [x.set_index(dup_list) for x in sheets]withdf = pd.concat(dfs, axis=1).reset_index()
– jezrael
Nov 25 '18 at 16:16
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
|
show 5 more comments
2
2
2
I believe you need set.intersection with mapped all columns names to sets:
df1 = pd.DataFrame(columns=list('acbd'))
df2 = pd.DataFrame(columns=list('abde'))
df3 = pd.DataFrame(columns=list('cbad'))
df4 = pd.DataFrame(columns=list('acbf'))
sheets = [df1, df2, df3, df4]
L = [x.columns for x in sheets]
#less readable
#L = [x for x in sheets]
dup_list = list(set.intersection(*map(set,L)))
print (dup_list)
['a', 'b']
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
I believe you need set.intersection with mapped all columns names to sets:
df1 = pd.DataFrame(columns=list('acbd'))
df2 = pd.DataFrame(columns=list('abde'))
df3 = pd.DataFrame(columns=list('cbad'))
df4 = pd.DataFrame(columns=list('acbf'))
sheets = [df1, df2, df3, df4]
L = [x.columns for x in sheets]
#less readable
#L = [x for x in sheets]
dup_list = list(set.intersection(*map(set,L)))
print (dup_list)
['a', 'b']
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
answered Nov 25 '18 at 15:25
jezraeljezrael
330k23271350
330k23271350
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in allsheetnames:( Or maybe some whitespaces? Is possible seeL = [x.columns for x in sheets]print (L)?
– jezrael
Nov 25 '18 at 15:42
1
So if testL = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets]what isprint (L)?
– jezrael
Nov 25 '18 at 16:04
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually bydf4.columns = ['cola1','col2',...]. And then better is useconcatlikemerge- Usedfs = [x.set_index(dup_list) for x in sheets]withdf = pd.concat(dfs, axis=1).reset_index()
– jezrael
Nov 25 '18 at 16:16
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
|
show 5 more comments
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in allsheetnames:( Or maybe some whitespaces? Is possible seeL = [x.columns for x in sheets]print (L)?
– jezrael
Nov 25 '18 at 15:42
1
So if testL = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets]what isprint (L)?
– jezrael
Nov 25 '18 at 16:04
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually bydf4.columns = ['cola1','col2',...]. And then better is useconcatlikemerge- Usedfs = [x.set_index(dup_list) for x in sheets]withdf = pd.concat(dfs, axis=1).reset_index()
– jezrael
Nov 25 '18 at 16:16
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
Hi @Jezrael, this is helpful thank you. However its returning an empty list
– Paul
Nov 25 '18 at 15:39
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in all
sheetnames :( Or maybe some whitespaces? Is possible see L = [x.columns for x in sheets] print (L) ?– jezrael
Nov 25 '18 at 15:42
@Paul - hmmm, try test only 2-3 DataFrames, in my opinion in real data not exist same columns in all
sheetnames :( Or maybe some whitespaces? Is possible see L = [x.columns for x in sheets] print (L) ?– jezrael
Nov 25 '18 at 15:42
1
1
So if test
L = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets] what is print (L) ?– jezrael
Nov 25 '18 at 16:04
So if test
L = [x.columns[x.columns.isin(['PAY REFERENCE', 'NAME'])] for x in sheets] what is print (L) ?– jezrael
Nov 25 '18 at 16:04
1
1
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually by
df4.columns = ['cola1','col2',...]. And then better is use concat like merge - Use dfs = [x.set_index(dup_list) for x in sheets] with df = pd.concat(dfs, axis=1).reset_index()– jezrael
Nov 25 '18 at 16:16
@Paul - In my opinion if need working with DataFrame with no header later, still is necessary set them - e.g. manually by
df4.columns = ['cola1','col2',...]. And then better is use concat like merge - Use dfs = [x.set_index(dup_list) for x in sheets] with df = pd.concat(dfs, axis=1).reset_index()– jezrael
Nov 25 '18 at 16:16
1
1
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
@jezreal, Thank you, I will look into it
– Paul
Nov 25 '18 at 16:29
|
show 5 more comments
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53468943%2fcollecting-duplicate-column-names-from-multiple-dataframes-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
