Scrapy - How to get duplicate request referer
When I turn on DUPEFILTER_DEBUG, I got:
2016-09-21 01:48:29 [scrapy] DEBUG: Filtered duplicate request: http://www.example.org/example.html>
The problem is, I need to know the duplicate request's referrer to debug the code. How can I debug the referrer?
python python-2.7 web-scraping scrapy scrapy-settings
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
add a comment |
When I turn on DUPEFILTER_DEBUG, I got:
2016-09-21 01:48:29 [scrapy] DEBUG: Filtered duplicate request: http://www.example.org/example.html>
The problem is, I need to know the duplicate request's referrer to debug the code. How can I debug the referrer?
python python-2.7 web-scraping scrapy scrapy-settings
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52
add a comment |
When I turn on DUPEFILTER_DEBUG, I got:
2016-09-21 01:48:29 [scrapy] DEBUG: Filtered duplicate request: http://www.example.org/example.html>
The problem is, I need to know the duplicate request's referrer to debug the code. How can I debug the referrer?
python python-2.7 web-scraping scrapy scrapy-settings
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
When I turn on DUPEFILTER_DEBUG, I got:
2016-09-21 01:48:29 [scrapy] DEBUG: Filtered duplicate request: http://www.example.org/example.html>
The problem is, I need to know the duplicate request's referrer to debug the code. How can I debug the referrer?
python python-2.7 web-scraping scrapy scrapy-settings
python python-2.7 web-scraping scrapy scrapy-settings
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
edited Nov 24 '18 at 14:24
Aminah Nuraini
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
asked Sep 21 '16 at 2:36
Aminah NurainiAminah Nuraini
6,56244562
6,56244562
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52
add a comment |
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52
add a comment |
1 Answer
1
active
oldest
votes
One option would be a custom filter based on the built-in RFPDupeFilter filter:
from scrapy.dupefilters import RFPDupeFilter
class MyDupeFilter(RFPDupeFilter):
def log(self, request, spider):
self.logger.debug(request.headers.get("REFERER"), extra={'spider': spider})
super(MyDupeFilter, self).log(request, spider)
Don't forget to set the DUPEFILTER_CLASS setting to point to your custom class.
(not tested)
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f39606643%2fscrapy-how-to-get-duplicate-request-referer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
One option would be a custom filter based on the built-in RFPDupeFilter filter:
from scrapy.dupefilters import RFPDupeFilter
class MyDupeFilter(RFPDupeFilter):
def log(self, request, spider):
self.logger.debug(request.headers.get("REFERER"), extra={'spider': spider})
super(MyDupeFilter, self).log(request, spider)
Don't forget to set the DUPEFILTER_CLASS setting to point to your custom class.
(not tested)
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
add a comment |
One option would be a custom filter based on the built-in RFPDupeFilter filter:
from scrapy.dupefilters import RFPDupeFilter
class MyDupeFilter(RFPDupeFilter):
def log(self, request, spider):
self.logger.debug(request.headers.get("REFERER"), extra={'spider': spider})
super(MyDupeFilter, self).log(request, spider)
Don't forget to set the DUPEFILTER_CLASS setting to point to your custom class.
(not tested)
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
add a comment |
One option would be a custom filter based on the built-in RFPDupeFilter filter:
from scrapy.dupefilters import RFPDupeFilter
class MyDupeFilter(RFPDupeFilter):
def log(self, request, spider):
self.logger.debug(request.headers.get("REFERER"), extra={'spider': spider})
super(MyDupeFilter, self).log(request, spider)
Don't forget to set the DUPEFILTER_CLASS setting to point to your custom class.
(not tested)
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
One option would be a custom filter based on the built-in RFPDupeFilter filter:
from scrapy.dupefilters import RFPDupeFilter
class MyDupeFilter(RFPDupeFilter):
def log(self, request, spider):
self.logger.debug(request.headers.get("REFERER"), extra={'spider': spider})
super(MyDupeFilter, self).log(request, spider)
Don't forget to set the DUPEFILTER_CLASS setting to point to your custom class.
(not tested)
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
answered Sep 21 '16 at 4:48
alecxealecxe
323k68623850
323k68623850
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f39606643%2fscrapy-how-to-get-duplicate-request-referer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Try implementing your own visited log, in memory or in file, by for example using a yielded links pipeline.
– Evhz
Sep 21 '16 at 8:52