How to locate button containing text with Selenium?
0
What I need: switch to the Reviews tab in description of an extension from Chrome Store (e.g. this one) in order to count the number of reviews.
What I've done: Used BeautifulSoup + Selenium to switch between tabs. I used driver.find_element_by_id('id') BUT it returns an error that it can not find the element.
Here's the code I use:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(url)
button = driver.find_element_by_id(':22')
button.click()
page = requests.get(driver.current_url)
soup = BeautifulSoup(page.content,'html5lib')
comment_list = soup.find('div', class_ = 'e-f-b-L') #the class of reviews I need to count.
Here's the html-code of the Review button element:
Issues:
How do I make it click the 'Reviews' button so the 'Reviews' tab is displayed?
selenium web-scraping beautifulsoup
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
add a comment |
0
What I need: switch to the Reviews tab in description of an extension from Chrome Store (e.g. this one) in order to count the number of reviews.
What I've done: Used BeautifulSoup + Selenium to switch between tabs. I used driver.find_element_by_id('id') BUT it returns an error that it can not find the element.
Here's the code I use:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(url)
button = driver.find_element_by_id(':22')
button.click()
page = requests.get(driver.current_url)
soup = BeautifulSoup(page.content,'html5lib')
comment_list = soup.find('div', class_ = 'e-f-b-L') #the class of reviews I need to count.
Here's the html-code of the Review button element:
Issues:
How do I make it click the 'Reviews' button so the 'Reviews' tab is displayed?
selenium web-scraping beautifulsoup
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41
add a comment |
0
0
0
What I need: switch to the Reviews tab in description of an extension from Chrome Store (e.g. this one) in order to count the number of reviews.
What I've done: Used BeautifulSoup + Selenium to switch between tabs. I used driver.find_element_by_id('id') BUT it returns an error that it can not find the element.
Here's the code I use:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(url)
button = driver.find_element_by_id(':22')
button.click()
page = requests.get(driver.current_url)
soup = BeautifulSoup(page.content,'html5lib')
comment_list = soup.find('div', class_ = 'e-f-b-L') #the class of reviews I need to count.
Here's the html-code of the Review button element:
Issues:
How do I make it click the 'Reviews' button so the 'Reviews' tab is displayed?
selenium web-scraping beautifulsoup
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
What I need: switch to the Reviews tab in description of an extension from Chrome Store (e.g. this one) in order to count the number of reviews.
What I've done: Used BeautifulSoup + Selenium to switch between tabs. I used driver.find_element_by_id('id') BUT it returns an error that it can not find the element.
Here's the code I use:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(url)
button = driver.find_element_by_id(':22')
button.click()
page = requests.get(driver.current_url)
soup = BeautifulSoup(page.content,'html5lib')
comment_list = soup.find('div', class_ = 'e-f-b-L') #the class of reviews I need to count.
Here's the html-code of the Review button element:
Issues:
How do I make it click the 'Reviews' button so the 'Reviews' tab is displayed?
selenium web-scraping beautifulsoup
selenium web-scraping beautifulsoup
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
edited Nov 23 '18 at 22:03
Corey Goldberg
36.8k22107122
36.8k22107122
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
asked Nov 23 '18 at 14:48
Sergei Shumilin
348
348
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41
add a comment |
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41
add a comment |
1 Answer
1
active
oldest
votes
1
You can click on that Reviews tab very smoothly If you define a simple xpath like '//div[.="Reviews"]' or so. Check out the script as a proof of concept:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
driver.quit()
To make it headless:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
print("It's done")
answered Nov 23 '18 at 19:52
SIM
10.2k3742
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this linedriver.quit(), so that you can see the change in the browser @Sergei Shumilin.
– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
|
show 2 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53448791%2fhow-to-locate-button-containing-text-with-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
You can click on that Reviews tab very smoothly If you define a simple xpath like '//div[.="Reviews"]' or so. Check out the script as a proof of concept:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
driver.quit()
To make it headless:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
print("It's done")
answered Nov 23 '18 at 19:52
SIM
10.2k3742
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this linedriver.quit(), so that you can see the change in the browser @Sergei Shumilin.
– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
|
show 2 more comments
1
You can click on that Reviews tab very smoothly If you define a simple xpath like '//div[.="Reviews"]' or so. Check out the script as a proof of concept:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
driver.quit()
To make it headless:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
print("It's done")
answered Nov 23 '18 at 19:52
SIM
10.2k3742
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this linedriver.quit(), so that you can see the change in the browser @Sergei Shumilin.
– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
|
show 2 more comments
1
1
1
You can click on that Reviews tab very smoothly If you define a simple xpath like '//div[.="Reviews"]' or so. Check out the script as a proof of concept:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
driver.quit()
To make it headless:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
print("It's done")
answered Nov 23 '18 at 19:52
SIM
10.2k3742
You can click on that Reviews tab very smoothly If you define a simple xpath like '//div[.="Reviews"]' or so. Check out the script as a proof of concept:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
driver.quit()
To make it headless:
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://chrome.google.com/webstore/detail/emoji-keyboard-by-emojion/ipdjnhgkpapgippgcgkfcbpdpcgifncb?hl=en"
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
wait = ui.WebDriverWait(driver, 10)
driver.get(url)
wait.until(lambda driver: driver.find_element_by_xpath('//div[.="Reviews"]')).click()
print("It's done")
answered Nov 23 '18 at 19:52
SIM
10.2k3742
edited Nov 27 '18 at 12:42
answered Nov 23 '18 at 19:52
SIM
10.2k3742
answered Nov 23 '18 at 19:52
SIM
10.2k3742
answered Nov 23 '18 at 19:52
SIM
10.2k3742
10.2k3742
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this linedriver.quit(), so that you can see the change in the browser @Sergei Shumilin.
– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
|
show 2 more comments
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this linedriver.quit(), so that you can see the change in the browser @Sergei Shumilin.
– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
Hi! Thank you for help! Does this code works for you?
– Sergei Shumilin
Nov 26 '18 at 8:53
1
1
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this line
driver.quit(), so that you can see the change in the browser @Sergei Shumilin.– SIM
Nov 26 '18 at 8:58
Sure it is. Why it comes to your mind that there is any doubdt. Check it out yourself. Make sure to comment out this line
driver.quit(), so that you can see the change in the browser @Sergei Shumilin.– SIM
Nov 26 '18 at 8:58
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
It works!) Thank you very much!) Can you tell two more things: 1) how to handle with pagination if we don't know the exact number of pages? 2) how to make the browser open in the background in order to save the processor resourses?
– Sergei Shumilin
Nov 27 '18 at 8:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
For your first question: If you wanna play with pagination, create a new post describing your requirement and drop here a link. I'll take a look. I could not understand your second question. Did you mean to run the browser headlessly?
– SIM
Nov 27 '18 at 9:00
1
1
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
Check out the edit @Sergei Shumilin.
– SIM
Nov 27 '18 at 12:43
|
show 2 more comments
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53448791%2fhow-to-locate-button-containing-text-with-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
That button doesn't do anything in terms of loading content. If you plan on using beautiful soup you can skip that part.
– pguardiario
Nov 24 '18 at 0:12
But the content is hidden before I push the buttom explicitly and BeautidulSoup can't find it. If it's not - I beg you to show how.
– Sergei Shumilin
Nov 25 '18 at 18:33
driver.find_element_by_class_name('classname').click() you can do this using class_name
– Ashfaque Marfani
Nov 27 '18 at 13:41