optimal sample size for control/test group for T-test
Recently we launched a feature on one of our website pages.I have all the historical data worth 6 months about the page including impressions and CTR .It has been 20 days since we launched the feature and now we want to know if there is any significant lift in CTR post-launch. Is there a way to determine how many impressions is statistically significant to conduct a T-test to check lift in CTR assuming page views before launch was control group and post launch is test group. How much historical data do I need to look at for control group and evaluate the required sample size for test group based on that. Any lead OR different approach is highly appreciated.
statistics sampling digital t-test hypothesis-test
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
add a comment |
Recently we launched a feature on one of our website pages.I have all the historical data worth 6 months about the page including impressions and CTR .It has been 20 days since we launched the feature and now we want to know if there is any significant lift in CTR post-launch. Is there a way to determine how many impressions is statistically significant to conduct a T-test to check lift in CTR assuming page views before launch was control group and post launch is test group. How much historical data do I need to look at for control group and evaluate the required sample size for test group based on that. Any lead OR different approach is highly appreciated.
statistics sampling digital t-test hypothesis-test
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21
add a comment |
Recently we launched a feature on one of our website pages.I have all the historical data worth 6 months about the page including impressions and CTR .It has been 20 days since we launched the feature and now we want to know if there is any significant lift in CTR post-launch. Is there a way to determine how many impressions is statistically significant to conduct a T-test to check lift in CTR assuming page views before launch was control group and post launch is test group. How much historical data do I need to look at for control group and evaluate the required sample size for test group based on that. Any lead OR different approach is highly appreciated.
statistics sampling digital t-test hypothesis-test
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
Recently we launched a feature on one of our website pages.I have all the historical data worth 6 months about the page including impressions and CTR .It has been 20 days since we launched the feature and now we want to know if there is any significant lift in CTR post-launch. Is there a way to determine how many impressions is statistically significant to conduct a T-test to check lift in CTR assuming page views before launch was control group and post launch is test group. How much historical data do I need to look at for control group and evaluate the required sample size for test group based on that. Any lead OR different approach is highly appreciated.
statistics sampling digital t-test hypothesis-test
statistics sampling digital t-test hypothesis-test
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
asked Nov 28 '18 at 22:09
M_NayakM_Nayak
12
12
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21
add a comment |
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53528860%2foptimal-sample-size-for-control-test-group-for-t-test%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53528860%2foptimal-sample-size-for-control-test-group-for-t-test%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Your terminology is confusing. What is CTR? What do you mean by significant lift? What are impressions?Are you trying to compare the mean (of something?) before and after this new feature (I am assuming because you mentioned t-tests)? What is this feature you are measuring and what type is it? 6 months should be more than enough if we are talking internet page views.
– user2974951
Nov 29 '18 at 9:11
Sorry for the confusion.CTR is click through rate. Impressions mean pageviews and clicks are the number of clicks to various links on the web page. We want to know if there is a statistically significant change in our CTR before and after feature launch. What I do not know is how to decide what is the sample size for my control and test group with confidence interval of 95%. In a A/B test, you show both variants to equal number of users at the same time. But in this case its before and after launch and thats where it gets complicated.
– M_Nayak
Nov 29 '18 at 14:21