Skip to content

Conversation

@hardbyte
Copy link
Collaborator

This calculation now takes into account the use of blocking. Also because it seemed somewhat non-trivial to compute I added caching in redis.

Closes #542

@hardbyte hardbyte requested a review from wilko77 April 17, 2020 05:49

log.debug("Updating redis cache for run")
set_run_state_active(run_id)
progress_cache.save_current_progress(comparisons=0, run_id=run_id)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was why I'd seen a log messages saving "0 comparisons".

Copy link
Collaborator

@wilko77 wilko77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Did you figure out why it was sometimes showing a progress of more than 100%?

@hardbyte
Copy link
Collaborator Author

Yeah it was just a dumb 2 party assumption on my side, I was first calculating the total number of comparisons as:

For each data provider multiple the size of each block together, then sum resulting number
of comparisons in each block all together.

When the actual calculation needed to be:

for each block:
        for each **pairwise combination** of data providers in each block:
                multiply the block sizes together
        then sum number of comparisons for each block
Sum the number of comparisons in all blocks together

@hardbyte hardbyte merged commit 23370ee into develop Apr 19, 2020
@hardbyte hardbyte deleted the fix-542 branch April 19, 2020 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Progress metrics not taking blocks into account

3 participants