#409 indexer optimizations #410
Conversation
proxy/indexer/indexer_base.py
Outdated
|
|
||
| if len(self.transaction_receipts) > 0: | ||
| self.min_known_tx = min(self.transaction_receipts) | ||
| self.max_known_tx = max(self.transaction_receipts) |
There was a problem hiding this comment.
Seems this call can cause unnecessary iteration over all table. I'd rather prefer to explicitely define methods min_key and max_key in SQLDictBinKey that will execute corresponding optimized queries in DB.
There was a problem hiding this comment.
It is impossible in the current realization of SQLDictBinKey. It stores the key as serialized binary.
There was a problem hiding this comment.
I suggest to separate current key column by two columns: 'slot', 'signature' and use it in composite key. The 'slot' column then can be used to create index. This will allow to create optimized queries for airdropper and also to find min and max txs
proxy/indexer/airdropper.py
Outdated
| if trx['transaction']['message']['instructions'] is not None: | ||
| self.process_trx_airdropper_mode(trx) | ||
| max_slot = 0 | ||
| for slot_sig, trx in sorted(self.transaction_receipts.iteritems(), reverse=True): |
There was a problem hiding this comment.
This call could be potentially very slow because it will always sort all available transactions. I'd suggest to implement special method in SqlDictBinKeys that would query all trxs after given slot ordered backwards
There was a problem hiding this comment.
It would be nice to have a metric, let's call it "process_receipts_ms"
What do you think of this?
* Create TrxReceiptsStorage * Use new container in indexer and airdropper * Fix bug * Fix assertions in tests, use random start_slot Co-authored-by: Ivan Loboda <[email protected]>
Add to
SQLDictclass ability to work with key-value storage with complex keys.In
indexer_baseuseSQLDictBinKeyto store known transactions. Use slot and signature pair as key. Save only transactions directed toneon-evm. And added function to move transactions from the old table to the new.Request from Solana transactions newer than in known transactions table.
In
indexerandairdroperuse ordered transactions directly from the known transactions table by sorting by slot and remembering the last processed slot to skip it on the next step.In
indexerif from a certain slot there is no unprocessed transaction skip those.