Skip to content

Merge develop into master for v1.13.0-beta2 #554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hardbyte merged 150 commits into from
Apr 30, 2020
Merged

Merge develop into master for v1.13.0-beta2 #554

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
02b7f95
Bump app version to 1.13.0-beta, frontend version to 1.4.6-beta
hardbyte Feb 10, 2020
881a6e1
Update worker autoscaler configuration
hardbyte Feb 10, 2020
21d20e0
Update changelog for 1.13.0-beta release
hardbyte Feb 10, 2020
81d6850
Feature openapi3 with blocking data upload support (#479)
wilko77 Feb 12, 2020
52fd307
Allow user to provide arbitrary config
hardbyte Feb 11, 2020
7688035
Allow user to provide arbitrary deployment annotations
hardbyte Feb 11, 2020
f508dee
Try use default tracing connection
hardbyte Feb 11, 2020
37182ee
Use default tracing connection for api too
hardbyte Feb 11, 2020
1ceea55
Extract load_yaml_config from setup_logging and add tests
hardbyte Feb 11, 2020
8c64217
K8s: Add a config file for tracing. Always mount all config files in …
hardbyte Feb 11, 2020
2cb9a3e
Remove redundant helper function from test_serialization
hardbyte Feb 11, 2020
7ccfda4
Remove individual tracing setting in favor of passing in a file path.
hardbyte Feb 11, 2020
d160540
Use tracing config file or default
hardbyte Feb 11, 2020
78b5603
Strings to yaml not objects
hardbyte Feb 12, 2020
ff633c8
Add config volume to db init job
hardbyte Feb 12, 2020
7938529
Try to use jaeger during CI testing for api
hardbyte Feb 12, 2020
0771973
Use default tracing connection for workers
hardbyte Feb 12, 2020
9a4c8bc
Try enable tracing
hardbyte Feb 12, 2020
f0481d5
Use default tracing connection for api too
hardbyte Feb 12, 2020
016a3c8
Ok enough of azure, just enable the tracing annotation by default
hardbyte Feb 12, 2020
d13eb79
Use default logging config in k8s
hardbyte Feb 12, 2020
1920a94
fix filename typo
hardbyte Feb 12, 2020
bac1025
cleanup after review
hardbyte Feb 12, 2020
381acc4
Only log celery container from worker pod
hardbyte Feb 12, 2020
ff024d4
Remove a stray print statement
hardbyte Feb 13, 2020
d39f530
Minor tweaks to improve logging
hardbyte Feb 13, 2020
34e7c68
Add docs on debugging locally (#504)
hardbyte Feb 19, 2020
88c3705
Feature extend binary format (#505)
wilko77 Feb 19, 2020
d5787ee
Refactor to use a separate base image in Docker
hardbyte Sep 26, 2019
7c6e4af
CI building of base image
hardbyte Feb 17, 2020
a95b6fb
Build Anonlink app using tag for base image
hardbyte Feb 17, 2020
a8195fe
Document base image
hardbyte Feb 17, 2020
6e23392
Update a few dependencies in base image
hardbyte Feb 17, 2020
e0b71a9
Update alpine to 3.11.3 and pin all c dependencies
hardbyte Feb 19, 2020
aad76e5
Update structlog
hardbyte Feb 19, 2020
b37ba6d
Update devops docs
hardbyte Feb 19, 2020
e381a10
Bump pytest from 5.1.3 to 5.3.5 in /base (#508)
dependabot-preview[bot] Feb 19, 2020
318fef1
Bump tornado from 4.5.3 to 6.0.3 in /base
dependabot-preview[bot] Feb 19, 2020
7a927a5
Detect invalid encoding size earlier (#507)
hardbyte Feb 20, 2020
b5a7705
Feature blocking (#510)
hardbyte Feb 23, 2020
296ee5a
Bump ijson from 2.3 to 2.6.1 in /base
dependabot-preview[bot] Feb 24, 2020
9c362c5
Update base image to use Python 3.8.2 (#518)
hardbyte Mar 1, 2020
3e62b93
CI: Skip building base image if not required
hardbyte Mar 1, 2020
40b5a33
CI system now automatically uses latest base image (#519)
hardbyte Mar 1, 2020
96c82dc
Store uploaded encodings in database
hardbyte Feb 24, 2020
b831449
Bulk upload encodings to database
hardbyte Feb 27, 2020
04576db
Clean up comments
hardbyte Feb 28, 2020
4f2f4b5
More efficient fetching from encodingblocks table
hardbyte Feb 28, 2020
cad57d1
Transaction size for encoding upload will depend on encoding_size.
hardbyte Mar 2, 2020
79a7e89
Peek at actual encoding size
hardbyte Mar 2, 2020
3b0905f
Bump redis from 3.2.1 to 3.4.1 in /base
dependabot-preview[bot] Mar 2, 2020
69c19c5
Add some integration tests that have access to the database
hardbyte Feb 27, 2020
2b16ddf
CI can run integration tests using docker-compose.
hardbyte Feb 28, 2020
34818e6
Insertion tests should also fetch data afterwards
hardbyte Feb 28, 2020
e69b411
Update docs regarding integration testing
hardbyte Mar 1, 2020
2a894b8
Basic integration tests for redis
hardbyte Mar 2, 2020
61459bf
Update integration tests following code review
hardbyte Mar 3, 2020
39cfe94
Configure jaeger for docker-compose
hardbyte Mar 3, 2020
1540c0f
Tweak to allow debugging of tests from docker-compose
hardbyte Mar 3, 2020
db51552
Transaction size for encoding upload will depend on encoding_size.
hardbyte Mar 2, 2020
e73597e
Pull encoding data from postgres
hardbyte Mar 2, 2020
1bdf1e8
User provided binary packed uploads now go straight into db
hardbyte Mar 3, 2020
0ff7bf0
Stop storing uploaded encodings in minio
hardbyte Mar 3, 2020
07e2371
Fixup tracing in comparison task
hardbyte Mar 3, 2020
c692bb2
Extract logging and span setup from views
hardbyte Mar 3, 2020
40dd27f
Handle null filename
hardbyte Mar 3, 2020
d5f3578
Minor cleanup to imports and unused functions
hardbyte Mar 3, 2020
631aa67
Add test to check stored encodings in db
hardbyte Mar 3, 2020
9c8304a
Fix local jaeger (#523)
hardbyte Mar 4, 2020
ecd6cfe
Remove timing requirements from insertion test
hardbyte Mar 15, 2020
23e78d4
Add docstring to precheck_encoding_upload function
hardbyte Mar 15, 2020
f8c32c3
Merge branch 'develop' into feature-use-encodings-from-db
hardbyte Mar 16, 2020
1a37280
Merge pull request #522 from data61/feature-use-encodings-from-db
hardbyte Mar 16, 2020
560e414
Update kubernetes deployment and CI (#529)
hardbyte Mar 18, 2020
4dd360a
Add function to fetch block ids and sizes from db
hardbyte Mar 4, 2020
3c6776b
Retrieve blocking info in create_comparison_jobs task
hardbyte Mar 4, 2020
8c96ca8
WIP - identify blocks that need to be broken up further
hardbyte Mar 4, 2020
2077fe2
Query for getting encodings in a block
hardbyte Mar 6, 2020
3002978
Split tasks into chunks using blocking information
hardbyte Mar 6, 2020
79bfcbf
Refactor create comparison jobs function
hardbyte Mar 8, 2020
5e18c12
More refactoring of chunk creation
hardbyte Mar 9, 2020
c1cc6eb
Add a few unit tests for chunking
hardbyte Mar 9, 2020
54460be
Add database index on encodings table
hardbyte Mar 9, 2020
0669620
clknblocks not clksnblocks and other minor cleanup
hardbyte Mar 10, 2020
e946e2c
cleanup
hardbyte Mar 10, 2020
4958d15
Add blocking concept to docs
hardbyte Mar 13, 2020
c260291
Deduplicate candidate pairs before solving
hardbyte Mar 15, 2020
37faa25
Catch the empty candidate pair case
hardbyte Mar 15, 2020
12525bc
Simplify solver task by using anonlink's _merge_similarities function
hardbyte Mar 16, 2020
9b8fcff
Update celery
hardbyte Mar 16, 2020
a3ab400
Address code review feedback
hardbyte Mar 18, 2020
5e123ff
Bump version to beta2
hardbyte Mar 18, 2020
1caa1a5
Celery concurrency defaults
hardbyte Mar 19, 2020
70794a5
Add another layer of tracing into the comparison task
hardbyte Mar 19, 2020
07d4291
Update task names in celery routing
hardbyte Mar 19, 2020
3e70f57
Faster encoding retrieval by using COPY.
hardbyte Mar 22, 2020
d6c8751
Pass on stored size when retrieving encodings from DB
hardbyte Mar 22, 2020
71e254d
Increase time on test
hardbyte Mar 22, 2020
50b6357
Refactor binary copy into own function for easier reuse and testing
hardbyte Mar 23, 2020
4c4d198
Add more detailed tracing around binary encoding insertions.
hardbyte Mar 24, 2020
7dbcf2d
Add tests for binary copy function
hardbyte Mar 24, 2020
5d94af8
Bump celery from 4.4.0 to 4.4.2 in /base
dependabot-preview[bot] Mar 24, 2020
00c715d
Split tests into unit and e2e
hardbyte Mar 11, 2020
b611b17
Add uses blocking to e2e test config
hardbyte Mar 12, 2020
b66b262
Build and use separate e2e test image on azure
hardbyte Mar 12, 2020
4d279c0
Remove duplicate config and utils from unit tests
hardbyte Mar 24, 2020
0ed4bc2
Cleanup docker compose (#533)
hardbyte Mar 26, 2020
cfe6549
Use local benchmark cache (#531)
hardbyte Mar 26, 2020
5130de7
Increase CELERYD_MAX_TASKS_PER_CHILD default (#534)
hardbyte Mar 27, 2020
446e3f9
Update tutorials to use anonlinkclient instead of clkhash (#536)
hardbyte Apr 6, 2020
f27e2d3
Bump flower from 0.9.3 to 0.9.4 in /base
dependabot-preview[bot] Apr 6, 2020
f69bbc4
Bump alpine from 3.11.3 to 3.11.5 in /base
dependabot-preview[bot] Apr 6, 2020
86a40d1
Update minio deployment (#539)
hardbyte Apr 8, 2020
bc66664
Bump flask from 1.1.1 to 1.1.2 in /base
dependabot-preview[bot] Apr 8, 2020
a994be4
Bump requests from 2.22.0 to 2.23.0 in /base
dependabot-preview[bot] Apr 13, 2020
012d9a0
Feature blocky benchmark (#541)
wilko77 Apr 17, 2020
23370ee
Fix calculation of total number of comparisons (#543)
hardbyte Apr 19, 2020
f17f336
Design doc for pull feature. (#537)
hardbyte Apr 19, 2020
ad5076e
Bump minio from 5.0.7 to 5.0.10 in /base
dependabot-preview[bot] Apr 20, 2020
4e2653d
Temporary object store credentials endpoint (#544)
hardbyte Apr 21, 2020
4298031
Use docker-compose without having to set the --project-directory (#547)
hardbyte Apr 22, 2020
e01a584
Expose minio (#548)
hardbyte Apr 27, 2020
52b9355
Modify upload endpoint to take information on external encodings.
hardbyte Apr 23, 2020
6902939
Minor: Fix api url in e2e test helper, ignore created log files when …
hardbyte Apr 23, 2020
33c5e98
Simplify abort_if_inconsistent_upload and add support for external en…
hardbyte Apr 23, 2020
efec190
Add utility functions to convert to common clknblocks format.
hardbyte Apr 23, 2020
91a52c8
Catch most common error in status check during startup
hardbyte Apr 23, 2020
edbc169
Modify upload endpoint to take external data, add background task to …
hardbyte Apr 23, 2020
fadc3cc
Use correct endpoint for testing object store uploads
hardbyte Apr 24, 2020
19240b6
Explicitly throw a notimplemented error for blocks + external encodings
hardbyte Apr 24, 2020
9356d41
Add test for pulling external block data from object store
hardbyte Apr 27, 2020
0feb5a1
Update OpenAPI to support external blocking info
hardbyte Apr 27, 2020
0b883ad
Extract object store path template
hardbyte Apr 27, 2020
1a3ab15
Task and backend support added for external blocks
hardbyte Apr 27, 2020
91d4599
Move object store credential parsing into object store module
hardbyte Apr 27, 2020
9b72ecd
Include release in k8s selector for workers
hardbyte Apr 29, 2020
8bc502f
Check for executable runs after handling data upload
hardbyte Apr 28, 2020
fbebda9
Upgrade to newer minio chart. Update open api doc
hardbyte Apr 29, 2020
50df1ff
Add to e2e upload test, update readme.
hardbyte Apr 29, 2020
602c2e6
Extend test using external blocking data to include a run
hardbyte Apr 29, 2020
a6ff398
Edit and add log statements throughout upload path
hardbyte Apr 29, 2020
191b9c6
Skip cleaning up object store files, when there are no files
hardbyte Apr 29, 2020
d2bad47
Fix tracing on project cleanup
hardbyte Apr 29, 2020
91b678f
Set the encoding size on upload
hardbyte Apr 29, 2020
0705541
Log expected and unexpected problems during upload with different sev…
hardbyte Apr 29, 2020
8c7cd8c
Remove uploaded data from object store during cleanup
hardbyte Apr 29, 2020
53ecdba
Update benchmark version
hardbyte Apr 30, 2020
b788ac4
Add changelog for v1.13.0-beta2
hardbyte Apr 30, 2020
7d12e4c
Remove deprecated debug flag from k8s config
hardbyte Apr 30, 2020
ade1124
Increase chance of a flaky test passing
hardbyte Apr 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix local jaeger (#523) 
* Configure jaeger for docker-compose
* Tweak to allow debugging of tests from docker-compose
  • Loading branch information
@hardbyte
hardbyte authored Mar 4, 2020
commit 9c8304ad3669aa582df6f3317c9a4fe15cbdd29a
3 changes: 2 additions & 1 deletion tools/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ services:
environment:
- SERVER=http://nginx:8851
- INITIAL_DELAY=20
entrypoint: /bin/sh -c "dockerize -wait tcp://nginx:8851/v1/status -timeout 1m python -m pytest -n 16 entityservice/tests --junitxml=testResults.xml -x"
- JAEGER_AGENT_HOST=jaeger
command: /bin/sh -c "dockerize -wait tcp://nginx:8851/v1/status -timeout 1m python -m pytest -n 4 entityservice/tests --junitxml=testResults.xml -x"
depends_on:
- db
- redis
Expand Down
2 changes: 2 additions & 0 deletions tools/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ services:
- MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
- FLASK_DB_MIN_CONNECTIONS=1
- FLASK_DB_MAX_CONNECTIONS=10
- JAEGER_AGENT_HOST=jaeger
depends_on:
- db
- db_init
Expand Down Expand Up @@ -73,6 +74,7 @@ services:
#- CHUNK_SIZE_AIM=300_000_000
- CELERY_DB_MIN_CONNECTIONS=1
- CELERY_DB_MAX_CONNECTIONS=3
- JAEGER_AGENT_HOST=jaeger


nginx:
Expand Down