Skip to content

feat: add AddSeq for incremental sync#644

Merged
cheggaaa merged 2 commits intomainfrom
seq
Mar 24, 2026
Merged

feat: add AddSeq for incremental sync#644
cheggaaa merged 2 commits intomainfrom
seq

Conversation

@requilence
Copy link
Copy Markdown
Contributor

Summary

  • Add per-space monotonically growing AddSeq counter to storage changes
  • Enables incremental sync between space storage and CRDT storage: a consumer tracks its last processed seq and fetches only newer changes via GetAfterAddSeq, without rescanning the full tree
  • Single atomic.Add(1) per AddAll/AddAllNoError call — all changes in one batch share the same seq
  • Shared atomic.Uint64 counter across all tree storages in a space, set by spaceStorage
  • spaceStorage.New initializes counter from max existing seq in DB on startup
  • Gaps in seq are expected and safe (rolled-back transactions, validator noop counters) — only monotonic growth is guaranteed
  • Seq is local to the peer, not synced across nodes

Changes

  • StorageChange.AddSeq field, AddSeqKey constant, Storage.GetAfterAddSeq interface method
  • storage.SetAddSeq / storageDeferredCreation.SetAddSeq wiring
  • spaceStorage manages shared counter, injects into all tree storages
  • objecttreevalidator uses a noop zero-counter for ephemeral validation storages

Test plan

  • TestAddSeq_AddAll — batch seq assignment, increment across calls
  • TestAddSeq_AddAllNoError — duplicate inserts tolerated, seq still increments
  • TestAddSeq_NoopCounter — zero-valued counter works (validator path)
  • TestAddSeq_GetAfterAddSeq — threshold queries return correct subsets
  • TestAddSeq_GetAfterAddSeq_Empty — no false positives on empty results
  • TestAddSeq_Parallel — 10 goroutines sharing one counter, no duplicate seqs
  • TestAddSeq_DeferredCreation — seq propagated through lazy storage creation

Implement WebTransport as a new transport option alongside yamux and quic.
Includes native server/client with TLS handshake, WASM/browser client
(dial-only), PeerService integration with optional transport registration,
and configuration support.
@requilence requilence force-pushed the seq branch 4 times, most recently from 43c4e66 to cb4ec5a Compare March 24, 2026 12:46
Add space-global monotonic add sequence (AddSeq) to storage changes,
enabling incremental sync by allowing queries for changes after a given
sequence number. Includes GetAfterAddSeq query method on storage and
per-space atomic counter managed by spaceStorage.
Base automatically changed from webtransport to main March 24, 2026 12:51
@requilence requilence requested a review from cheggaaa March 24, 2026 12:56
@cheggaaa cheggaaa merged commit 6b2f9d6 into main Mar 24, 2026
3 checks passed
@cheggaaa cheggaaa deleted the seq branch March 24, 2026 12:58
@github-actions github-actions bot locked and limited conversation to collaborators Mar 24, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants