Is my Robots.txt correctly configured with Magento 2?












0















In Magento 2 version 2.2.2 navigate to content -> Configuration. Make sure that default robots are set to index > follow. Add the following to the custom instructions:




User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml




When I visit: https://www.example.com/robots.txt I get the below:
enter image description here



My question is how to ensure that what I enter as custom instructions are applied to the robots.txt? Further, when I check in google webmaster tool I can see that Google has the same robots.txt in the above picture. How do search engines read the custom instructions and were those really applied? Where are they?










share|improve this question



























    0















    In Magento 2 version 2.2.2 navigate to content -> Configuration. Make sure that default robots are set to index > follow. Add the following to the custom instructions:




    User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml




    When I visit: https://www.example.com/robots.txt I get the below:
    enter image description here



    My question is how to ensure that what I enter as custom instructions are applied to the robots.txt? Further, when I check in google webmaster tool I can see that Google has the same robots.txt in the above picture. How do search engines read the custom instructions and were those really applied? Where are they?










    share|improve this question

























      0












      0








      0








      In Magento 2 version 2.2.2 navigate to content -> Configuration. Make sure that default robots are set to index > follow. Add the following to the custom instructions:




      User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml




      When I visit: https://www.example.com/robots.txt I get the below:
      enter image description here



      My question is how to ensure that what I enter as custom instructions are applied to the robots.txt? Further, when I check in google webmaster tool I can see that Google has the same robots.txt in the above picture. How do search engines read the custom instructions and were those really applied? Where are they?










      share|improve this question














      In Magento 2 version 2.2.2 navigate to content -> Configuration. Make sure that default robots are set to index > follow. Add the following to the custom instructions:




      User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml




      When I visit: https://www.example.com/robots.txt I get the below:
      enter image description here



      My question is how to ensure that what I enter as custom instructions are applied to the robots.txt? Further, when I check in google webmaster tool I can see that Google has the same robots.txt in the above picture. How do search engines read the custom instructions and were those really applied? Where are they?







      magento2 robots.txt google-webmaster-tools






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Dec 31 '17 at 7:58









      samsam

      619




      619
























          1 Answer
          1






          active

          oldest

          votes


















          0














          By using these custom instructions you allow and disallow Google from indexing the Magneto store URLs of your choice. Yes, your custom instructions will be applied.



          To know more about it and its configuration you can refer this guide which can clear your confusions: How to Setup Magento 2 Robots.txt






          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48040919%2fis-my-robots-txt-correctly-configured-with-magento-2%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0














            By using these custom instructions you allow and disallow Google from indexing the Magneto store URLs of your choice. Yes, your custom instructions will be applied.



            To know more about it and its configuration you can refer this guide which can clear your confusions: How to Setup Magento 2 Robots.txt






            share|improve this answer




























              0














              By using these custom instructions you allow and disallow Google from indexing the Magneto store URLs of your choice. Yes, your custom instructions will be applied.



              To know more about it and its configuration you can refer this guide which can clear your confusions: How to Setup Magento 2 Robots.txt






              share|improve this answer


























                0












                0








                0







                By using these custom instructions you allow and disallow Google from indexing the Magneto store URLs of your choice. Yes, your custom instructions will be applied.



                To know more about it and its configuration you can refer this guide which can clear your confusions: How to Setup Magento 2 Robots.txt






                share|improve this answer













                By using these custom instructions you allow and disallow Google from indexing the Magneto store URLs of your choice. Yes, your custom instructions will be applied.



                To know more about it and its configuration you can refer this guide which can clear your confusions: How to Setup Magento 2 Robots.txt







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 28 '18 at 10:23









                Syed Muneeb Ul HasanSyed Muneeb Ul Hasan

                18714




                18714
































                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48040919%2fis-my-robots-txt-correctly-configured-with-magento-2%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

                    Calculate evaluation metrics using cross_val_predict sklearn

                    Insert data from modal to MySQL (multiple modal on website)