Skip to content

Comparison tasks now based on blocks #527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hardbyte merged 27 commits into from
Mar 24, 2020
Merged

Comparison tasks now based on blocks #527

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
065b11a
Add function to fetch block ids and sizes from db
hardbyte Mar 4, 2020
6f7c75c
Retrieve blocking info in create_comparison_jobs task
hardbyte Mar 4, 2020
ebcb248
WIP - identify blocks that need to be broken up further
hardbyte Mar 4, 2020
a066ccc
Query for getting encodings in a block
hardbyte Mar 6, 2020
fb550de
Split tasks into chunks using blocking information
hardbyte Mar 6, 2020
610b3bb
Refactor create comparison jobs function
hardbyte Mar 8, 2020
d838fe4
More refactoring of chunk creation
hardbyte Mar 9, 2020
ec36e8d
Add a few unit tests for chunking
hardbyte Mar 9, 2020
ddcbcc3
Add database index on encodings table
hardbyte Mar 9, 2020
4ab16e6
clknblocks not clksnblocks and other minor cleanup
hardbyte Mar 10, 2020
d66bf58
cleanup
hardbyte Mar 10, 2020
1e5151f
Add blocking concept to docs
hardbyte Mar 13, 2020
aec9b5c
Deduplicate candidate pairs before solving
hardbyte Mar 15, 2020
f30c819
Catch the empty candidate pair case
hardbyte Mar 15, 2020
9dc59e1
Simplify solver task by using anonlink's _merge_similarities function
hardbyte Mar 16, 2020
6219e44
Update celery
hardbyte Mar 16, 2020
0b6a4c2
Address code review feedback
hardbyte Mar 18, 2020
5467362
Bump version to beta2
hardbyte Mar 18, 2020
f342d5a
Celery concurrency defaults
hardbyte Mar 19, 2020
2add5ef
Add another layer of tracing into the comparison task
hardbyte Mar 19, 2020
e2ebe99
Update task names in celery routing
hardbyte Mar 19, 2020
38b624f
Faster encoding retrieval by using COPY.
hardbyte Mar 22, 2020
7ec7fef
Pass on stored size when retrieving encodings from DB
hardbyte Mar 22, 2020
24caa79
Increase time on test
hardbyte Mar 22, 2020
dc1983b
Refactor binary copy into own function for easier reuse and testing
hardbyte Mar 23, 2020
8bae410
Add more detailed tracing around binary encoding insertions.
hardbyte Mar 24, 2020
88e968d
Add tests for binary copy function
hardbyte Mar 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Celery concurrency defaults
  • Loading branch information
@hardbyte
hardbyte committed Mar 19, 2020
commit f342d5ad5aca4098760ec71c98fe28ea2e035825
3 changes: 2 additions & 1 deletion backend/entityservice/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ class Config(object):

CELERYD_PREFETCH_MULTIPLIER = int(os.getenv('CELERYD_PREFETCH_MULTIPLIER', '1'))
CELERYD_MAX_TASKS_PER_CHILD = int(os.getenv('CELERYD_MAX_TASKS_PER_CHILD', '4'))
CELERYD_CONCURRENCY = int(os.getenv("CELERYD_CONCURRENCY", '0'))
# number of concurrent worker processes/threads, executing tasks
CELERYD_CONCURRENCY = int(os.getenv("CELERYD_CONCURRENCY", '2'))
CELERY_ACKS_LATE = os.getenv('CELERY_ACKS_LATE', 'false') == 'true'

# Number of comparisons per chunk (on average).
Expand Down
2 changes: 1 addition & 1 deletion tools/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ services:
- MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
- CELERY_ACKS_LATE=true
- REDIS_USE_SENTINEL=false
- CELERYD_MAX_TASKS_PER_CHILD=30
- CELERYD_MAX_TASKS_PER_CHILD=2048
#- CHUNK_SIZE_AIM=300_000_000
- CELERY_DB_MIN_CONNECTIONS=1
- CELERY_DB_MAX_CONNECTIONS=3
Expand Down