Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Added <deltafetch_passthrough> request metaparameter.
This allows for more fine-tuned applicability of deltafetch.
Even if deltafetch wants to skip a request, setting this parameter will force it to go through.
Example:
scrapy.Request(url=myUrl, meta={'deltafetch_passthrough':True})
  • Loading branch information
ulrichschmid committed Nov 7, 2016
commit 928bd4fb76863de55f7b45a316a441f2006dd256
2 changes: 1 addition & 1 deletion scrapy_deltafetch/middleware.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def spider_closed(self, spider):

def process_spider_output(self, response, result, spider):
for r in result:
if isinstance(r, Request):
if isinstance(r, Request) and not r.meta.get('deltafetch_passthrough') == True:
key = self._get_key(r)
if self.db.has_key(key):
logger.info("Ignoring already visited: %s" % r)
Expand Down