Skip to content

[core change!] replace bsddb3 with default Python dbm #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 20, 2021
Merged

[core change!] replace bsddb3 with default Python dbm #40

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c9dd29e
[core change!] replace bsddb3 with default Python dbm
pawelmhm Jun 11, 2021
0c647a8
[docs, setup.py] cleaning up after bsddb3 removal
pawelmhm Sep 17, 2021
ad8b7db
[tests] add benchmark
pawelmhm Sep 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[docs, setup.py] cleaning up after bsddb3 removal
  • Loading branch information
@pawelmhm
pawelmhm committed Sep 17, 2021
commit 0c647a87e866ab16abe7750df0491bbdb9ed6781
6 changes: 6 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
Changes
=======

2.0.0 (2021-09-20)
------------------
* drop Python 2 support
* replace bsddb3 with Python's dbm for storing request data
* [dev workflow] changed from Travis to Github Actions

1.2.1 (2017-02-09)
------------------

Expand Down
13 changes: 4 additions & 9 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,16 @@ scrapy-deltafetch
:target: https://codecov.io/gh/scrapy-plugins/scrapy-deltafetch

This is a Scrapy spider middleware to ignore requests
to pages containing items seen in previous crawls of the same spider,
thus producing a "delta crawl" containing only new items.
to pages seen in previous crawls of the same spider,
thus producing a "delta crawl" containing only new requests.

This also speeds up the crawl, by reducing the number of requests that need
to be crawled, and processed (typically, item requests are the most CPU
intensive).

Requirements
============

DeltaFetch middleware depends on Python's bsddb3_ package.

On Ubuntu/Debian, you may need to install ``libdb-dev`` if it's not installed already.
DeltaFetch middleware uses Python's dbm_ package to store requests fingerprints.

.. _bsddb3: https://pypi.python.org/pypi/bsddb3
.. _dbm: https://docs.python.org/3/library/dbm.html


Installation
Expand Down
8 changes: 5 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@
'License :: OSI Approved :: BSD License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
],
install_requires=['Scrapy>=1.1.0', 'bsddb3']
install_requires=['Scrapy>=1.1.0']
)