How to scrape a web site using Python, Requests and Xpath?
I try to scrape first name + last name of people on this web page (https://www.meleenumerique.com/scientist_comite) using the code below but it doesn't work. How can I determine what's wrong with it?
This is the code I wrote
from lxml import html
import csv,os,json
import requests
url="https://www.meleenumerique.com/scientist_comite"
r=requests.get(url)
t=html.fromstring(r.content)
title=t.xpath('/html/head/title/text()')
#Create the list of speaker
speaker=t.xpath('//span[contains(@class,"speaker-name")]//text()')
print(title)
print("Speakers:",speaker)
python web-scraping python-requests lxml
edited Nov 27 '18 at 20:13
halfer
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
add a comment |
I try to scrape first name + last name of people on this web page (https://www.meleenumerique.com/scientist_comite) using the code below but it doesn't work. How can I determine what's wrong with it?
This is the code I wrote
from lxml import html
import csv,os,json
import requests
url="https://www.meleenumerique.com/scientist_comite"
r=requests.get(url)
t=html.fromstring(r.content)
title=t.xpath('/html/head/title/text()')
#Create the list of speaker
speaker=t.xpath('//span[contains(@class,"speaker-name")]//text()')
print(title)
print("Speakers:",speaker)
python web-scraping python-requests lxml
edited Nov 27 '18 at 20:13
halfer
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04
add a comment |
I try to scrape first name + last name of people on this web page (https://www.meleenumerique.com/scientist_comite) using the code below but it doesn't work. How can I determine what's wrong with it?
This is the code I wrote
from lxml import html
import csv,os,json
import requests
url="https://www.meleenumerique.com/scientist_comite"
r=requests.get(url)
t=html.fromstring(r.content)
title=t.xpath('/html/head/title/text()')
#Create the list of speaker
speaker=t.xpath('//span[contains(@class,"speaker-name")]//text()')
print(title)
print("Speakers:",speaker)
python web-scraping python-requests lxml
edited Nov 27 '18 at 20:13
halfer
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
I try to scrape first name + last name of people on this web page (https://www.meleenumerique.com/scientist_comite) using the code below but it doesn't work. How can I determine what's wrong with it?
This is the code I wrote
from lxml import html
import csv,os,json
import requests
url="https://www.meleenumerique.com/scientist_comite"
r=requests.get(url)
t=html.fromstring(r.content)
title=t.xpath('/html/head/title/text()')
#Create the list of speaker
speaker=t.xpath('//span[contains(@class,"speaker-name")]//text()')
print(title)
print("Speakers:",speaker)
python web-scraping python-requests lxml
python web-scraping python-requests lxml
edited Nov 27 '18 at 20:13
halfer
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
edited Nov 27 '18 at 20:13
halfer
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
edited Nov 27 '18 at 20:13
halfer
14.4k758109
edited Nov 27 '18 at 20:13
halfer
14.4k758109
edited Nov 27 '18 at 20:13
halfer
14.4k758109
14.4k758109
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
asked Nov 24 '18 at 16:01
Nico2806Nico2806
184
184
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04
add a comment |
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04
add a comment |
1 Answer
1
active
oldest
votes
You can try with this Requests-HTML library which should let you scrape the content from that page. This library supports xpath and has the ability to take care of dynamic content.
import requests_html
session = requests_html.HTMLSession()
r = session.get('https://www.meleenumerique.com/scientist_comite')
r.html.render(sleep=5, timeout=8)
for item in r.html.xpath("//*[contains(@class,'speaker-name')]"):
print(item.text)
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53459924%2fhow-to-scrape-a-web-site-using-python-requests-and-xpath%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can try with this Requests-HTML library which should let you scrape the content from that page. This library supports xpath and has the ability to take care of dynamic content.
import requests_html
session = requests_html.HTMLSession()
r = session.get('https://www.meleenumerique.com/scientist_comite')
r.html.render(sleep=5, timeout=8)
for item in r.html.xpath("//*[contains(@class,'speaker-name')]"):
print(item.text)
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
add a comment |
You can try with this Requests-HTML library which should let you scrape the content from that page. This library supports xpath and has the ability to take care of dynamic content.
import requests_html
session = requests_html.HTMLSession()
r = session.get('https://www.meleenumerique.com/scientist_comite')
r.html.render(sleep=5, timeout=8)
for item in r.html.xpath("//*[contains(@class,'speaker-name')]"):
print(item.text)
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
add a comment |
You can try with this Requests-HTML library which should let you scrape the content from that page. This library supports xpath and has the ability to take care of dynamic content.
import requests_html
session = requests_html.HTMLSession()
r = session.get('https://www.meleenumerique.com/scientist_comite')
r.html.render(sleep=5, timeout=8)
for item in r.html.xpath("//*[contains(@class,'speaker-name')]"):
print(item.text)
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
You can try with this Requests-HTML library which should let you scrape the content from that page. This library supports xpath and has the ability to take care of dynamic content.
import requests_html
session = requests_html.HTMLSession()
r = session.get('https://www.meleenumerique.com/scientist_comite')
r.html.render(sleep=5, timeout=8)
for item in r.html.xpath("//*[contains(@class,'speaker-name')]"):
print(item.text)
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
answered Nov 25 '18 at 8:03
SIMSIM
10.3k3743
10.3k3743
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53459924%2fhow-to-scrape-a-web-site-using-python-requests-and-xpath%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Possible duplicate of Web-scraping JavaScript page with Python
– Turtvaiz
Nov 24 '18 at 23:04