Skip to content

#409 indexer optimizations #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
otselnik merged 10 commits into from
Dec 30, 2021
Merged

#409 indexer optimizations #410

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
47c2c2e
#409 indexer optomizations
otselnik Dec 25, 2021
884b82a
Merge remote-tracking branch 'origin/develop' into 409_indexer_optimi…
otselnik Dec 25, 2021
496e4da
#409 catch exceptiopns
otselnik Dec 25, 2021
4036918
#409 add airdropper and faucet artifacts
otselnik Dec 26, 2021
3e1e825
#409 remove unused variables
otselnik Dec 26, 2021
84d2f8a
#409 return`self.last_slot` to `IndexerBase` class
otselnik Dec 26, 2021
097e839
#409 merge SQLDict and SQLDictBinKey classes to one
otselnik Dec 26, 2021
224f5c2
#409 remove debug log
otselnik Dec 26, 2021
560c4d6
#409 add metrics
otselnik Dec 27, 2021
799285c
421 optimise indexer transaction receipt storage (#423)
ivandzen Dec 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
#409 remove unused variables
  • Loading branch information
@otselnik
otselnik committed Dec 26, 2021
commit 3e1e825473031a960d95193c9cc84de3db563c58
3 changes: 0 additions & 3 deletions proxy/indexer/indexer_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,12 @@ def __init__(self,
self.evm_loader_id = evm_loader_id
self.client = Client(solana_url)
self.transaction_receipts = SQLDictBinKey(tablename="known_transactions")
self.last_slot = start_slot
self.current_slot = 0
self.counter_ = 0

if len(self.transaction_receipts) > 0:
self.min_known_tx = min(self.transaction_receipts)
self.max_known_tx = max(self.transaction_receipts)
Copy link
Contributor

@ivandzen ivandzen Dec 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this call can cause unnecessary iteration over all table. I'd rather prefer to explicitely define methods min_key and max_key in SQLDictBinKey that will execute corresponding optimized queries in DB.

Copy link
Contributor Author

@otselnik otselnik Dec 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is impossible in the current realization of SQLDictBinKey. It stores the key as serialized binary.

Copy link
Contributor

@ivandzen ivandzen Dec 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to separate current key column by two columns: 'slot', 'signature' and use it in composite key. The 'slot' column then can be used to create index. This will allow to create optimized queries for airdropper and also to find min and max txs

else:
self.min_known_tx = (0, None)
self.max_known_tx = (0, None)

self._move_data_from_old_table()
Expand Down