why does z-test indicate significantly different for 2 distributions that looks similar (hypothesis testing)
0
I have two distributions below (Kaggle dataset: Rossman sales) that look similar visually: Sales on normal days & sales on school holiday.
However, they seems to fail z-test (hypothesis testing) in Python - why is that so?
How should I perform the statistical test (z-test) in Python? Should I use pooled or unequalvar (should I use same variance or different)? I also found out that switching School_hol_sales and Normal_day_sales in the code below yield different results and I am not sure why.
School_hol_sales = df[(df.Open==1)&(df.SchoolHoliday==1)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
Normal_day_sales = df[(df.Open==1)&(df.SchoolHoliday==0)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
School_hol_sales.mean(), Normal_day_sales.mean() # (6230.4, 5904.6)
School_hol_sales.std(), Normal_day_sales.std() # (2841.8, 2602.9)
# which is the correct one?
import statsmodels.stats.api as sms
cm = sms.CompareMeans(sms.DescrStatsW(School_hol_sales), sms.DescrStatsW(Normal_day_sales))
z, pval = cm.ztest_ind(alternative='larger', usevar='unequal')
print('z: {} , pval: {}'.format(z, pval))
from statsmodels.stats.weightstats import ztest
z, pval = ztest(School_hol_sales,Normal_day_sales, alternative='larger', usevar='pooled', ddof=1.0)
print('z: {} , pval: {}'.format(z, pval))
Output:
z: 28.53350149055591 , pval: 2.2504631945823565e-179
z: 30.17089944207645 , pval: 2.853425122518376e-200
python statistics data-science hypothesis-test
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
add a comment |
0
I have two distributions below (Kaggle dataset: Rossman sales) that look similar visually: Sales on normal days & sales on school holiday.
However, they seems to fail z-test (hypothesis testing) in Python - why is that so?
How should I perform the statistical test (z-test) in Python? Should I use pooled or unequalvar (should I use same variance or different)? I also found out that switching School_hol_sales and Normal_day_sales in the code below yield different results and I am not sure why.
School_hol_sales = df[(df.Open==1)&(df.SchoolHoliday==1)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
Normal_day_sales = df[(df.Open==1)&(df.SchoolHoliday==0)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
School_hol_sales.mean(), Normal_day_sales.mean() # (6230.4, 5904.6)
School_hol_sales.std(), Normal_day_sales.std() # (2841.8, 2602.9)
# which is the correct one?
import statsmodels.stats.api as sms
cm = sms.CompareMeans(sms.DescrStatsW(School_hol_sales), sms.DescrStatsW(Normal_day_sales))
z, pval = cm.ztest_ind(alternative='larger', usevar='unequal')
print('z: {} , pval: {}'.format(z, pval))
from statsmodels.stats.weightstats import ztest
z, pval = ztest(School_hol_sales,Normal_day_sales, alternative='larger', usevar='pooled', ddof=1.0)
print('z: {} , pval: {}'.format(z, pval))
Output:
z: 28.53350149055591 , pval: 2.2504631945823565e-179
z: 30.17089944207645 , pval: 2.853425122518376e-200
python statistics data-science hypothesis-test
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
1
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28
add a comment |
0
0
0
I have two distributions below (Kaggle dataset: Rossman sales) that look similar visually: Sales on normal days & sales on school holiday.
However, they seems to fail z-test (hypothesis testing) in Python - why is that so?
How should I perform the statistical test (z-test) in Python? Should I use pooled or unequalvar (should I use same variance or different)? I also found out that switching School_hol_sales and Normal_day_sales in the code below yield different results and I am not sure why.
School_hol_sales = df[(df.Open==1)&(df.SchoolHoliday==1)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
Normal_day_sales = df[(df.Open==1)&(df.SchoolHoliday==0)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
School_hol_sales.mean(), Normal_day_sales.mean() # (6230.4, 5904.6)
School_hol_sales.std(), Normal_day_sales.std() # (2841.8, 2602.9)
# which is the correct one?
import statsmodels.stats.api as sms
cm = sms.CompareMeans(sms.DescrStatsW(School_hol_sales), sms.DescrStatsW(Normal_day_sales))
z, pval = cm.ztest_ind(alternative='larger', usevar='unequal')
print('z: {} , pval: {}'.format(z, pval))
from statsmodels.stats.weightstats import ztest
z, pval = ztest(School_hol_sales,Normal_day_sales, alternative='larger', usevar='pooled', ddof=1.0)
print('z: {} , pval: {}'.format(z, pval))
Output:
z: 28.53350149055591 , pval: 2.2504631945823565e-179
z: 30.17089944207645 , pval: 2.853425122518376e-200
python statistics data-science hypothesis-test
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
I have two distributions below (Kaggle dataset: Rossman sales) that look similar visually: Sales on normal days & sales on school holiday.
However, they seems to fail z-test (hypothesis testing) in Python - why is that so?
How should I perform the statistical test (z-test) in Python? Should I use pooled or unequalvar (should I use same variance or different)? I also found out that switching School_hol_sales and Normal_day_sales in the code below yield different results and I am not sure why.
School_hol_sales = df[(df.Open==1)&(df.SchoolHoliday==1)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
Normal_day_sales = df[(df.Open==1)&(df.SchoolHoliday==0)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
School_hol_sales.mean(), Normal_day_sales.mean() # (6230.4, 5904.6)
School_hol_sales.std(), Normal_day_sales.std() # (2841.8, 2602.9)
# which is the correct one?
import statsmodels.stats.api as sms
cm = sms.CompareMeans(sms.DescrStatsW(School_hol_sales), sms.DescrStatsW(Normal_day_sales))
z, pval = cm.ztest_ind(alternative='larger', usevar='unequal')
print('z: {} , pval: {}'.format(z, pval))
from statsmodels.stats.weightstats import ztest
z, pval = ztest(School_hol_sales,Normal_day_sales, alternative='larger', usevar='pooled', ddof=1.0)
print('z: {} , pval: {}'.format(z, pval))
Output:
z: 28.53350149055591 , pval: 2.2504631945823565e-179
z: 30.17089944207645 , pval: 2.853425122518376e-200
python statistics data-science hypothesis-test
python statistics data-science hypothesis-test
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
asked Nov 28 '18 at 10:29
jasonlcy91jasonlcy91
779
779
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
1
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28
add a comment |
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
1
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
1
1
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53517313%2fwhy-does-z-test-indicate-significantly-different-for-2-distributions-that-looks%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53517313%2fwhy-does-z-test-indicate-significantly-different-for-2-distributions-that-looks%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution?
– DavidPM
Nov 28 '18 at 16:34
1
Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there.
– Robert Dodier
Nov 28 '18 at 19:28