Significantly lower accuracy while using class_weight for imbalanced dataset in keras












0















I have rather unintuitive problem. I am doing the sentiment analysis on Amazon Book reviews and the data set is heavily imbalanced. Positive reviews are almost 10 times the negative reviews, accuracy for both training and testing are around 90% (with imbalanced dataset). However, when I try to balance the dataset with the help of class_weight = {0:10 , 1:1} both training and testing accuracy drops to around 65%. Again if I do class_weight = {0:1 , 1:10} accuracy booms again, so apparently I am setting the class_weight wrong but as I understood because the number of positive reviews(1) are 10 times the number of negative reviews(0), shouldn't the class_weight be set as {0:10 , 1:1} ?



This is how I classify training and testing data:



x_train, x_test, y_train, y_test = train_test_split(sequences,labels, test_size = 0.33, random_state = 42)


This is my model:



model = Sequential()

model.add(Embedding(max_words, embedding_dim))

model.add(Dropout(0.5))

model.add(LSTM(32, dropout=0.5, recurrent_dropout=0.5))

model.add(Dense(2, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy',metrics=['acc'])

model.fit(x_train, y_train, epochs=10, batch_size=320, class_weight = {0:1 , 1:10})





python tensorflow keras sentiment-analysis







share|improve this question























  • Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

    – Dinari
    Nov 25 '18 at 11:57
















0















I have rather unintuitive problem. I am doing the sentiment analysis on Amazon Book reviews and the data set is heavily imbalanced. Positive reviews are almost 10 times the negative reviews, accuracy for both training and testing are around 90% (with imbalanced dataset). However, when I try to balance the dataset with the help of class_weight = {0:10 , 1:1} both training and testing accuracy drops to around 65%. Again if I do class_weight = {0:1 , 1:10} accuracy booms again, so apparently I am setting the class_weight wrong but as I understood because the number of positive reviews(1) are 10 times the number of negative reviews(0), shouldn't the class_weight be set as {0:10 , 1:1} ?



This is how I classify training and testing data:



x_train, x_test, y_train, y_test = train_test_split(sequences,labels, test_size = 0.33, random_state = 42)


This is my model:



model = Sequential()

model.add(Embedding(max_words, embedding_dim))

model.add(Dropout(0.5))

model.add(LSTM(32, dropout=0.5, recurrent_dropout=0.5))

model.add(Dense(2, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy',metrics=['acc'])

model.fit(x_train, y_train, epochs=10, batch_size=320, class_weight = {0:1 , 1:10})





python tensorflow keras sentiment-analysis







share|improve this question























  • Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

    – Dinari
    Nov 25 '18 at 11:57














0












0








0








I have rather unintuitive problem. I am doing the sentiment analysis on Amazon Book reviews and the data set is heavily imbalanced. Positive reviews are almost 10 times the negative reviews, accuracy for both training and testing are around 90% (with imbalanced dataset). However, when I try to balance the dataset with the help of class_weight = {0:10 , 1:1} both training and testing accuracy drops to around 65%. Again if I do class_weight = {0:1 , 1:10} accuracy booms again, so apparently I am setting the class_weight wrong but as I understood because the number of positive reviews(1) are 10 times the number of negative reviews(0), shouldn't the class_weight be set as {0:10 , 1:1} ?



This is how I classify training and testing data:



x_train, x_test, y_train, y_test = train_test_split(sequences,labels, test_size = 0.33, random_state = 42)


This is my model:



model = Sequential()

model.add(Embedding(max_words, embedding_dim))

model.add(Dropout(0.5))

model.add(LSTM(32, dropout=0.5, recurrent_dropout=0.5))

model.add(Dense(2, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy',metrics=['acc'])

model.fit(x_train, y_train, epochs=10, batch_size=320, class_weight = {0:1 , 1:10})





python tensorflow keras sentiment-analysis







share|improve this question














I have rather unintuitive problem. I am doing the sentiment analysis on Amazon Book reviews and the data set is heavily imbalanced. Positive reviews are almost 10 times the negative reviews, accuracy for both training and testing are around 90% (with imbalanced dataset). However, when I try to balance the dataset with the help of class_weight = {0:10 , 1:1} both training and testing accuracy drops to around 65%. Again if I do class_weight = {0:1 , 1:10} accuracy booms again, so apparently I am setting the class_weight wrong but as I understood because the number of positive reviews(1) are 10 times the number of negative reviews(0), shouldn't the class_weight be set as {0:10 , 1:1} ?



This is how I classify training and testing data:



x_train, x_test, y_train, y_test = train_test_split(sequences,labels, test_size = 0.33, random_state = 42)


This is my model:



model = Sequential()

model.add(Embedding(max_words, embedding_dim))

model.add(Dropout(0.5))

model.add(LSTM(32, dropout=0.5, recurrent_dropout=0.5))

model.add(Dense(2, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy',metrics=['acc'])

model.fit(x_train, y_train, epochs=10, batch_size=320, class_weight = {0:1 , 1:10})





python tensorflow keras sentiment-analysis




python tensorflow keras sentiment-analysis






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 25 '18 at 11:47









BlueMangoBlueMango

256




256













  • Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

    – Dinari
    Nov 25 '18 at 11:57



















  • Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

    – Dinari
    Nov 25 '18 at 11:57

















Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

– Dinari
Nov 25 '18 at 11:57





Note that one of the things the model learns is 'how probable' a sample is to be 0 or 1, even without seeing the sample. When you use class weights like that you actually change the model.

– Dinari
Nov 25 '18 at 11:57












2 Answers
2






active

oldest

votes


















1














Of course if you do not balance the loss you'll get better accuracy than if you balance it. Actually this is the reason for balancing. Otherwise the model that predict only positive class for all reviews will give you 90% accuracy. But this model is useless. Accuracy is generally bad metric for such strongly unbalanced datasets. Use F1 instead and you'll see that unbalanced model gives much worse F1.






share|improve this answer

































    0














    Setting higher class weight for a class with lower frequency in a data set is the right approach. More than Accuracy you can look at other more useful metrics like Precision, Recall, F1 score, auc_roc score (Concordance), Confusion matrix to actually understand what the model is learning.






    share|improve this answer
























    • okay but I want to understand about the significantly lower accuracy with class_weight

      – BlueMango
      Nov 25 '18 at 14:09











    • Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

      – AI_Learning
      Nov 25 '18 at 16:22











    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467109%2fsignificantly-lower-accuracy-while-using-class-weight-for-imbalanced-dataset-in%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    Of course if you do not balance the loss you'll get better accuracy than if you balance it. Actually this is the reason for balancing. Otherwise the model that predict only positive class for all reviews will give you 90% accuracy. But this model is useless. Accuracy is generally bad metric for such strongly unbalanced datasets. Use F1 instead and you'll see that unbalanced model gives much worse F1.






    share|improve this answer






























      1














      Of course if you do not balance the loss you'll get better accuracy than if you balance it. Actually this is the reason for balancing. Otherwise the model that predict only positive class for all reviews will give you 90% accuracy. But this model is useless. Accuracy is generally bad metric for such strongly unbalanced datasets. Use F1 instead and you'll see that unbalanced model gives much worse F1.






      share|improve this answer




























        1












        1








        1







        Of course if you do not balance the loss you'll get better accuracy than if you balance it. Actually this is the reason for balancing. Otherwise the model that predict only positive class for all reviews will give you 90% accuracy. But this model is useless. Accuracy is generally bad metric for such strongly unbalanced datasets. Use F1 instead and you'll see that unbalanced model gives much worse F1.






        share|improve this answer















        Of course if you do not balance the loss you'll get better accuracy than if you balance it. Actually this is the reason for balancing. Otherwise the model that predict only positive class for all reviews will give you 90% accuracy. But this model is useless. Accuracy is generally bad metric for such strongly unbalanced datasets. Use F1 instead and you'll see that unbalanced model gives much worse F1.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 25 '18 at 15:11

























        answered Nov 25 '18 at 14:30









        Andrey Kite GorinAndrey Kite Gorin

        16112




        16112

























            0














            Setting higher class weight for a class with lower frequency in a data set is the right approach. More than Accuracy you can look at other more useful metrics like Precision, Recall, F1 score, auc_roc score (Concordance), Confusion matrix to actually understand what the model is learning.






            share|improve this answer
























            • okay but I want to understand about the significantly lower accuracy with class_weight

              – BlueMango
              Nov 25 '18 at 14:09











            • Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

              – AI_Learning
              Nov 25 '18 at 16:22
















            0














            Setting higher class weight for a class with lower frequency in a data set is the right approach. More than Accuracy you can look at other more useful metrics like Precision, Recall, F1 score, auc_roc score (Concordance), Confusion matrix to actually understand what the model is learning.






            share|improve this answer
























            • okay but I want to understand about the significantly lower accuracy with class_weight

              – BlueMango
              Nov 25 '18 at 14:09











            • Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

              – AI_Learning
              Nov 25 '18 at 16:22














            0












            0








            0







            Setting higher class weight for a class with lower frequency in a data set is the right approach. More than Accuracy you can look at other more useful metrics like Precision, Recall, F1 score, auc_roc score (Concordance), Confusion matrix to actually understand what the model is learning.






            share|improve this answer













            Setting higher class weight for a class with lower frequency in a data set is the right approach. More than Accuracy you can look at other more useful metrics like Precision, Recall, F1 score, auc_roc score (Concordance), Confusion matrix to actually understand what the model is learning.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Nov 25 '18 at 12:32









            AI_LearningAI_Learning

            3,3112933




            3,3112933













            • okay but I want to understand about the significantly lower accuracy with class_weight

              – BlueMango
              Nov 25 '18 at 14:09











            • Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

              – AI_Learning
              Nov 25 '18 at 16:22



















            • okay but I want to understand about the significantly lower accuracy with class_weight

              – BlueMango
              Nov 25 '18 at 14:09











            • Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

              – AI_Learning
              Nov 25 '18 at 16:22

















            okay but I want to understand about the significantly lower accuracy with class_weight

            – BlueMango
            Nov 25 '18 at 14:09





            okay but I want to understand about the significantly lower accuracy with class_weight

            – BlueMango
            Nov 25 '18 at 14:09













            Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

            – AI_Learning
            Nov 25 '18 at 16:22





            Lower accuracy because of multiple reasons. Change your model architecture, data preprocessing, model hyper parameters, etc.

            – AI_Learning
            Nov 25 '18 at 16:22


















            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467109%2fsignificantly-lower-accuracy-while-using-class-weight-for-imbalanced-dataset-in%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

            Image
            0 I found a lot of questions abount appendices and ToC. Many users want appendices to be grouped in an Appendix part, however some problems arise with ToC, hyperref, PDF viewer bookmarks, and so on. There are different solutions which require extra packages, command patching and other extra code, however none of them satisfies me. I almost found an easy way to accomplish a good result, where appendices are added to bookmarks in the right way and hyperref links point to the right page. However, the number of the "Appendix" part page is wrong (it's the number of appendix A). Is there any EASY way to fix that? This is a MWE: documentclass{book} usepackage[nottoc,notlot,notlof]{tocbibind} usepackage{hyperref} begin{document} frontmatter tableofcontents mainmatter part{First} chapter{...
            Read more

            Calculate evaluation metrics using cross_val_predict sklearn

            Image
            1 In the sklearn.model_selection.cross_val_predict page it is stated: Generate cross-validated estimates for each input data point. It is not appropriate to pass these predictions into an evaluation metric. Can someone explain what does it mean? If this gives estimate of Y (y prediction) for every Y (true Y), why can't I calculate metrics such as RMSE or coefficient of determination using these results? python scikit-learn cross-validation share | improve this question edited Nov 28 '18 at 17:52 desertnaut 20.3k 7 43 79 ...
            Read more

            Insert data from modal to MySQL (multiple modal on website)

            Image
            0 I have problem with inserting data to MySQL from modal. My modal: <a href="#" class="badge badge-pill badge-success">6 komentarzy</a> <a href="#" class="badge badge-pill badge-danger">brak komentarzy</a> <a data-toggle="modal" href="#add_desk_comm_{$desk_['desk_id']}" data-target="#add_desk_comm_{$desk_['desk_id']}" class="ediiit">(dodaj)</a> <div class="modal fade" id="add_desk_comm_{$desk_['desk_id']}" tabindex="-1" role="dialog" aria-labelledby="edit_printer" aria-hidden="true"> <div class="modal-dialog" role="document"> ...
            Read more