Python - Browser Automation - MechanicalSoup/ BeautifulSoup
I am attempting to automate a button push that prompts javascript on my own website(godaddy server). I am running a .py from my MAC OS Terminal.
I have found beautiful soup and mechancial soup but only found documentation for parsing text or prefilling forms.
I have attempted mechanical soup code absent any 'form' mention, and merely trying to click a button based on a css selector.
I have played around with this code for a few hours and am not convinced what I want to accomplish is possible. Can anyone confirm this is possible with either of these modules? If not, what is a better tool?
I have just been using the example provided here: https://mechanicalsoup.readthedocs.io/en/stable/tutorial.html#first-contact-step-by-step
my code in a function:
def updatePrices()
br = mechanicalsoup.StatefulBrowser()
br.open("http://example.com")
br.get_current_page().find('button', id='exporter_decisionStream')
br.submit_selected()
updatePrices()
Any guidance is appreciated.
Thanks.
python browser automation mechanicalsoup
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
add a comment |
I am attempting to automate a button push that prompts javascript on my own website(godaddy server). I am running a .py from my MAC OS Terminal.
I have found beautiful soup and mechancial soup but only found documentation for parsing text or prefilling forms.
I have attempted mechanical soup code absent any 'form' mention, and merely trying to click a button based on a css selector.
I have played around with this code for a few hours and am not convinced what I want to accomplish is possible. Can anyone confirm this is possible with either of these modules? If not, what is a better tool?
I have just been using the example provided here: https://mechanicalsoup.readthedocs.io/en/stable/tutorial.html#first-contact-step-by-step
my code in a function:
def updatePrices()
br = mechanicalsoup.StatefulBrowser()
br.open("http://example.com")
br.get_current_page().find('button', id='exporter_decisionStream')
br.submit_selected()
updatePrices()
Any guidance is appreciated.
Thanks.
python browser automation mechanicalsoup
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35
add a comment |
I am attempting to automate a button push that prompts javascript on my own website(godaddy server). I am running a .py from my MAC OS Terminal.
I have found beautiful soup and mechancial soup but only found documentation for parsing text or prefilling forms.
I have attempted mechanical soup code absent any 'form' mention, and merely trying to click a button based on a css selector.
I have played around with this code for a few hours and am not convinced what I want to accomplish is possible. Can anyone confirm this is possible with either of these modules? If not, what is a better tool?
I have just been using the example provided here: https://mechanicalsoup.readthedocs.io/en/stable/tutorial.html#first-contact-step-by-step
my code in a function:
def updatePrices()
br = mechanicalsoup.StatefulBrowser()
br.open("http://example.com")
br.get_current_page().find('button', id='exporter_decisionStream')
br.submit_selected()
updatePrices()
Any guidance is appreciated.
Thanks.
python browser automation mechanicalsoup
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
I am attempting to automate a button push that prompts javascript on my own website(godaddy server). I am running a .py from my MAC OS Terminal.
I have found beautiful soup and mechancial soup but only found documentation for parsing text or prefilling forms.
I have attempted mechanical soup code absent any 'form' mention, and merely trying to click a button based on a css selector.
I have played around with this code for a few hours and am not convinced what I want to accomplish is possible. Can anyone confirm this is possible with either of these modules? If not, what is a better tool?
I have just been using the example provided here: https://mechanicalsoup.readthedocs.io/en/stable/tutorial.html#first-contact-step-by-step
my code in a function:
def updatePrices()
br = mechanicalsoup.StatefulBrowser()
br.open("http://example.com")
br.get_current_page().find('button', id='exporter_decisionStream')
br.submit_selected()
updatePrices()
Any guidance is appreciated.
Thanks.
python browser automation mechanicalsoup
python browser automation mechanicalsoup
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
asked Nov 28 '18 at 23:59
meganerdbluemeganerdblue
12
12
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35
add a comment |
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53529930%2fpython-browser-automation-mechanicalsoup-beautifulsoup%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53529930%2fpython-browser-automation-mechanicalsoup-beautifulsoup%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I'm not exactly sure what you mean by "prompts javascript", but MechanicalSoup doesn't interact with javascript, so that might be the problem. A tool like Selenium is better suited for javascript tasks. This MechanicalSoup FAQ says much the same thing.
– Daniel Hemberger
Nov 29 '18 at 0:29
Too clarify, from my understanding a button push in a form, would submit the form. Why can I not, interact with a button, that does not have a form attached to it? My button just calls a javascript function, this function should be executed on the web server and not by the python script, the script just pushes a button, the java should be run on the site as normal; my godaddy server.
– meganerdblue
Nov 29 '18 at 0:51
It's a good question! In pure HTML, a form tells the browser where to submit the form data (in this case, "the data" is the name of the button you pressed). MechanicalSoup parses that information from the html and mimics pressing the button. However, if there is no form and javascript that runs when the button is pressed is telling the browser what data to submit, then that's something that MechanicalSoup doesn't support (since it doesn't run javascript).
– Daniel Hemberger
Nov 30 '18 at 1:21
Thank you for coming back to this and helping a noob. I will look into Selenium.
– meganerdblue
Nov 30 '18 at 19:35