Skip to content

#764 Decrease the storage usage for Solana receipts #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
otselnik merged 13 commits into from
May 20, 2022
Merged

#764 Decrease the storage usage for Solana receipts #769

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1508550
Decrease the storage usage for Solana receipts neonlabsorg/proxy-mode…
otselnik May 16, 2022
6a7de1f
remove unused tables
otselnik May 16, 2022
a07f4af
optimize proccess spawning
otselnik May 16, 2022
1c82620
debug info
otselnik May 16, 2022
000333b
clearing list fix
otselnik May 16, 2022
ed1d5c0
fix "until" transaction
otselnik May 18, 2022
5f11060
do not store signatures with same slot
otselnik May 18, 2022
cfc3b0e
use `tx_list` without converting
otselnik May 18, 2022
836e7ec
update maximum_tx on end of loop
otselnik May 19, 2022
443e603
Merge remote-tracking branch 'origin/develop' into 764_do_not_store_s…
otselnik May 19, 2022
9562809
get rid of until option
otselnik May 20, 2022
5bc4bd5
optimizations
otselnik May 20, 2022
45bd265
Merge branch 'develop' into 764_do_not_store_solana_receipts_in_indexer
otselnik May 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
optimize proccess spawning
  • Loading branch information
@otselnik
otselnik committed May 16, 2022
commit a07f4af993858e367c8f7e2406781805b8e310ca
15 changes: 11 additions & 4 deletions proxy/indexer/indexer_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ def get_tx_receipts(self, stop_slot=None):
tx_list.append(signature)
if len(tx_list) >= 20:
poll_txs.append(tx_list)
poll_txs.append(tx_list)
if len(tx_list) > 0:
poll_txs.append(tx_list)
self._get_txs(poll_txs)

for signature in signatures:
Expand Down Expand Up @@ -172,11 +173,17 @@ def _get_signatures(self, before: Optional[str], until: Optional[str], limit: in
return result

def _get_txs(self, poll_txs: List[List[str]]) -> None:
pool = ThreadPool(PARALLEL_REQUESTS)
pool.map(self._get_tx_receipts, poll_txs)
poll_txs.clear()
self.debug(f'{len(poll_txs)}')
if len(poll_txs) > 1:
pool = ThreadPool(min(PARALLEL_REQUESTS, len(poll_txs)))
pool.map(self._get_tx_receipts, poll_txs)
poll_txs.clear()
else:
if len(poll_txs) > 0:
self._get_tx_receipts(poll_txs[0])

def _get_tx_receipts(self, sign_list: List[str]) -> None:
self.debug(f'{len(sign_list)}')
if len(sign_list) == 0:
return

Expand Down