Skip to content

#748 High memory usage in indexer on gathering txs from Solana and parsing txs #751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
afalaleev merged 11 commits into from
Apr 17, 2022
Merged

#748 High memory usage in indexer on gathering txs from Solana and parsing txs #751

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7a5ce87
When `poll_txs` gathered `INDEXER_POLL_COUNT` txs call gathering and …
otselnik Apr 15, 2022
3c7bf01
Limit get_txs output
otselnik Apr 15, 2022
f8f223a
Merge branch 'develop' into 748_indexer_mem_usage
otselnik Apr 15, 2022
98be48c
do not skip clots
otselnik Apr 15, 2022
e900ff7
Merge branch 'develop' into 748_indexer_mem_usage
otselnik Apr 15, 2022
2790d7f
Select limited range of transaction receipts. #748
afalaleev Apr 17, 2022
a703c56
Remove old holders from indexer cache
afalaleev Apr 14, 2022
40da019
Remove old holders from indexer cache
afalaleev Apr 14, 2022
494f663
Fix select query in get_txs. #748
afalaleev Apr 17, 2022
b7e9b99
Minimize log output
afalaleev Apr 17, 2022
62d6c2c
Fix
afalaleev Apr 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Select limited range of transaction receipts. #748
  • Loading branch information
@afalaleev
afalaleev committed Apr 17, 2022
commit 2790d7f07d2d3cf6e0ba6f6752d686902d2fcb96
40 changes: 28 additions & 12 deletions proxy/indexer/trx_receipts_storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,18 +45,34 @@ def contains(self, slot, signature):

def get_txs(self, start_slot=0):
with self._conn.cursor() as cur:
cur.execute(f'SELECT slot FROM {self._table_name}' +
f' WHERE slot >= {start_slot} ORDER BY slot ASC, tx_idx DESC' +
f' OFFSET {INDEXER_RECEIPTS_COUNT_LIMIT} LIMIT 1')
slot_row = cur.fetchone()

stop_slot_req_part = ''
if slot_row is not None:
stop_slot_req_part = f' AND slot <= {slot_row[0]}'

cur.execute(f'SELECT slot, signature, tx FROM {self._table_name}' +
f' WHERE slot >= {start_slot}' + stop_slot_req_part +
f' ORDER BY slot ASC, tx_idx DESC')
cur.execute(f'''
SELECT MIN(slot) FROM {self._table_name}' +
WHERE slot > %s'
''',
(start_slot,))
min_slot_row = cur.fetchone()
min_slot = (min_slot_row[0] if min_slot_row else 0)

cur.execute(f'''
SELECT MAX(t.slot) FROM (
SELECT slot FROM {self._table_name}
WHERE slot > %s
ORDER BY slot
LIMIT {INDEXER_RECEIPTS_COUNT_LIMIT}
) AS t
''',
(start_slot,))
limit_slot_row = cur.fetchone()
limit_slot = (limit_slot_row[0] if limit_slot_row else 0)

stop_slot = max(min_slot, limit_slot, start_slot + 1)

cur.execute(f'''
SELECT slot, signature, tx FROM {self._table_name}
WHERE slot >= %s AND slot <= %s
ORDER BY slot ASC, tx_idx DESC
''',
(start_slot, stop_slot,))
rows = cur.fetchall()

for row in rows:
Expand Down