Skip to content

Comparison tasks now based on blocks #527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hardbyte merged 27 commits into from
Mar 24, 2020
Merged

Comparison tasks now based on blocks #527

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
065b11a
Add function to fetch block ids and sizes from db
hardbyte Mar 4, 2020
6f7c75c
Retrieve blocking info in create_comparison_jobs task
hardbyte Mar 4, 2020
ebcb248
WIP - identify blocks that need to be broken up further
hardbyte Mar 4, 2020
a066ccc
Query for getting encodings in a block
hardbyte Mar 6, 2020
fb550de
Split tasks into chunks using blocking information
hardbyte Mar 6, 2020
610b3bb
Refactor create comparison jobs function
hardbyte Mar 8, 2020
d838fe4
More refactoring of chunk creation
hardbyte Mar 9, 2020
ec36e8d
Add a few unit tests for chunking
hardbyte Mar 9, 2020
ddcbcc3
Add database index on encodings table
hardbyte Mar 9, 2020
4ab16e6
clknblocks not clksnblocks and other minor cleanup
hardbyte Mar 10, 2020
d66bf58
cleanup
hardbyte Mar 10, 2020
1e5151f
Add blocking concept to docs
hardbyte Mar 13, 2020
aec9b5c
Deduplicate candidate pairs before solving
hardbyte Mar 15, 2020
f30c819
Catch the empty candidate pair case
hardbyte Mar 15, 2020
9dc59e1
Simplify solver task by using anonlink's _merge_similarities function
hardbyte Mar 16, 2020
6219e44
Update celery
hardbyte Mar 16, 2020
0b6a4c2
Address code review feedback
hardbyte Mar 18, 2020
5467362
Bump version to beta2
hardbyte Mar 18, 2020
f342d5a
Celery concurrency defaults
hardbyte Mar 19, 2020
2add5ef
Add another layer of tracing into the comparison task
hardbyte Mar 19, 2020
e2ebe99
Update task names in celery routing
hardbyte Mar 19, 2020
38b624f
Faster encoding retrieval by using COPY.
hardbyte Mar 22, 2020
7ec7fef
Pass on stored size when retrieving encodings from DB
hardbyte Mar 22, 2020
24caa79
Increase time on test
hardbyte Mar 22, 2020
dc1983b
Refactor binary copy into own function for easier reuse and testing
hardbyte Mar 23, 2020
8bae410
Add more detailed tracing around binary encoding insertions.
hardbyte Mar 24, 2020
88e968d
Add tests for binary copy function
hardbyte Mar 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Bump version to beta2
  • Loading branch information
@hardbyte
hardbyte committed Mar 18, 2020
commit 5467362eac837c06529e06ba393cfcb33ea25f83
2 changes: 1 addition & 1 deletion backend/entityservice/VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v1.13.0-beta
v1.13.0-beta2
2 changes: 1 addition & 1 deletion deployment/entity-service/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: entity-service
appVersion: 1.13.0-beta
appVersion: 1.13.0-beta2
version: 1.13.1
description: Privacy preserving record linkage service
sources:
Expand Down
14 changes: 5 additions & 9 deletions deployment/entity-service/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ api:

image:
repository: data61/anonlink-nginx
tag: "v1.4.6-beta"
tag: "v1.4.6-beta2"
# pullPolicy: Always
pullPolicy: IfNotPresent

Expand All @@ -55,7 +55,7 @@ api:
app:
image:
repository: data61/anonlink-app
tag: "v1.13.0-beta"
tag: "v1.13.0-beta2"
# pullPolicy: IfNotPresent
pullPolicy: Always

Expand All @@ -78,7 +78,7 @@ api:
## It cannot be updated! So we have a separate image + tag
image:
repository: data61/anonlink-app
tag: "v1.13.0-beta"
tag: "v1.13.0-beta2"

## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
resources:
Expand Down Expand Up @@ -149,7 +149,7 @@ workers:

image:
repository: "data61/anonlink-app"
tag: "v1.13.0-beta"
tag: "v1.13.0-beta2"
pullPolicy: Always

## The initial number of workers for this deployment
Expand Down Expand Up @@ -204,9 +204,7 @@ workers:
## our tasks are usually quite "long".
PREFETCH_MULTIPLIER: "1"
## Maximum number of tasks a pool worker process can execute before it’s replaced with a new one
## Low number is recommended, otherwise the celery workers may exhaust the available memory and threads.
## Cf issue https://github.com/data61/anonlink-entity-service/issues/410
MAX_TASKS_PER_CHILD: "30"
MAX_TASKS_PER_CHILD: "2048"
## Late ack means the task messages will be acknowledged after the task has been executed, not just before.
ACKS_LATE: "true"

Expand Down Expand Up @@ -262,8 +260,6 @@ postgresql:


global:
global:
storageClass: "default"

postgresql:
postgresqlDatabase: postgres
Expand Down
6 changes: 6 additions & 0 deletions docs/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ Next Version
------------


Version 1.13.0-beta2
-------------------

- Encodings are now stored in Postgres database instead of files in an object store.
- Initial support for user supplied blocking implemented.

Version 1.13.0-beta
-------------------

Expand Down
2 changes: 1 addition & 1 deletion docs/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
# The short X.Y version.
version = '1.13'
# The full version, including alpha/beta/rc tags.
release = '1.13.0-beta'
release = '1.13.0-beta2'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down
2 changes: 1 addition & 1 deletion frontend/VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v1.4.6-beta
v1.4.6-beta2