Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b8a3bae
feat(storage): storage-agnostic BlobStore layer + push-latency quick …
kevincodex1 Jun 22, 2026
9016e25
fix(storage): address CodeRabbit review (data-integrity + robustness)
kevincodex1 Jun 23, 2026
7b93d40
fix(storage): address beardthelion review (lock soundness, durability…
kevincodex1 Jun 23, 2026
b357510
fix(storage): isolate advisory-lock pool + #[must_use] write guard
kevincodex1 Jun 24, 2026
242c593
fix(storage): propagate durability errors + serialize background uploads
kevincodex1 Jun 24, 2026
f17fee0
fix(storage): address final review round + multithreaded compression
kevincodex1 Jul 23, 2026
d30af0d
fix(storage): address CodeRabbit round — key backslash rejection, loc…
kevincodex1 Jul 23, 2026
8c4ec8e
fix(storage): pending-upload marker + review round fixes
kevincodex1 Jul 23, 2026
75060ea
fix(storage): versioned pending marker, claim-first creation, fs etag…
kevincodex1 Jul 23, 2026
f154768
docs(storage): ObjectMeta etag doc reflects the fs sidecar etag
kevincodex1 Jul 23, 2026
5605b9c
fix(storage): harden marker machinery — durable pre-ack, locked diver…
kevincodex1 Jul 24, 2026
3cbcb2b
fix(storage): keep the marker gauge live; stripe the fs publish lock
kevincodex1 Jul 24, 2026
7e9fde0
fix(storage): in-flight etag provenance closes the upload-to-clear cr…
kevincodex1 Jul 25, 2026
1d5b479
fix(storage): content-validated recovery, locked creation lifecycle, …
kevincodex1 Jul 26, 2026
b014ce7
merge: main into feat/storage-abstraction (port storage abstraction o…
kevincodex1 Aug 31, 2026
9ce694b
fix(storage): path-traversal barrier for sibling paths; restore the F…
kevincodex1 Aug 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(storage): content-validated recovery, locked creation lifecycle, …
…dual lock keys

jatmn round on 7e9fde0 (all seven findings):

- Recovery validates heal candidates by CONTENT, not etag: it fetches
  the remote archive and compares its MD5 against the marker's recorded
  in-flight hash. This is backend-independent — correct under IPFS CIDs
  and SSE-KMS etags, and immune to the fs backend crashing between its
  sidecar and blob renames (stale bytes under a fresh etag now read as
  divergence, never as a completed upload). The fs backend's etag is
  the persisted content MD5, so a failed publish also restores the
  previous sidecar.
- record_inflight_upload is fallible and aborts the PUT when the intent
  cannot be durably recorded — an upload must never outrun the marker
  that explains it.
- Issue/close/merge handlers verify the pending marker actually exists
  before treating a failed release as recoverable; with no marker the
  request fails instead of acking an unprotected mutation.
- Creation and fork publication run inside RepoStore::create_published,
  which holds the per-repo advisory lock for the whole claim-to-
  publication lifecycle: a push through the just-visible row serializes
  behind publication, so failure compensation can never destroy a
  concurrently accepted push. release_after_write is gone.
- The ipfs backend is explicitly restricted to a single shared Kubo
  instance (MFS is daemon-local): startup warning + module/config/README
  docs state the deployment requirement.
- Writers acquire BOTH the new SHA-256 advisory-lock key and the legacy
  DefaultHasher key (all-or-nothing per attempt, partial grabs released
  before retrying), so writer exclusion holds across a mixed-binary
  rolling deploy; the legacy key drops once the fleet is upgraded.
  • Loading branch information
kevincodex1 committed Jul 26, 2026
commit 1d5b479a101f99ab8c07b2efedd553deaa1accaf
4 changes: 3 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ GITLAWB_REPOS_DIR=/data/repos
# s3 when a bucket is set, else fs when GITLAWB_STORAGE_FS_DIR is set, else
# local-only (repos live only on this node's disk). `ipfs` is never
# auto-selected — setting GITLAWB_IPFS_API alone keeps its pinning-only
# meaning; opt in explicitly with GITLAWB_STORAGE_BACKEND=ipfs.
# meaning; opt in explicitly with GITLAWB_STORAGE_BACKEND=ipfs. NOTE: the
# ipfs backend stores archives in the Kubo daemon's LOCAL MFS namespace, so
# every node must point at the SAME Kubo instance.
GITLAWB_STORAGE_BACKEND=
# Bucket for the s3 backend (Tigris, R2, AWS S3, MinIO, B2).
# GITLAWB_TIGRIS_BUCKET is honored as a legacy alias.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ Important node settings:
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
| `GITLAWB_STORAGE_BACKEND` | Object-storage backend for repo archives: `s3`, `fs`, or `ipfs`. Empty = auto-detect (`s3` if a bucket is set, else `fs` if a dir is set, else local-only). `ipfs` is never auto-selected. |
| `GITLAWB_STORAGE_BACKEND` | Object-storage backend for repo archives: `s3`, `fs`, or `ipfs`. Empty = auto-detect (`s3` if a bucket is set, else `fs` if a dir is set, else local-only). `ipfs` is never auto-selected and requires all nodes to share one Kubo instance (MFS is daemon-local). |
| `GITLAWB_S3_BUCKET` | Bucket for the `s3` backend (Tigris, R2, AWS S3, MinIO, B2). |
| `GITLAWB_S3_ENDPOINT` | Endpoint URL override for the `s3` backend (R2/MinIO; empty on Tigris/Fly). |
| `GITLAWB_S3_FORCE_PATH_STYLE` | Force path-style S3 addressing (MinIO and some S3-compatibles). |
Expand Down
46 changes: 35 additions & 11 deletions crates/gitlawb-node/src/api/issues.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,27 @@ pub async fn create_issue(
let release_result = guard.release(create_result.is_ok()).await;

create_result.map_err(|e| AppError::Git(e.to_string()))?;
// The git mutation is committed locally and protected by the pending-upload
// marker, so a durable-upload failure here is recoverable (the next
// successful upload re-syncs storage) — NOT a request failure. Failing the
// request would make the caller retry a non-idempotent mutation: the retry
// mints a new issue UUID and both issues eventually publish.
// A durable-upload failure is recoverable ONLY while the pending-upload
// marker protects the committed mutation (the next successful upload
// re-syncs storage). Verify the marker actually exists before choosing to
// succeed: if the marker write itself also failed, nothing protects the
// mutation from a stale-archive rollback, and the request must fail.
// (Succeeding here avoids non-idempotent retries: a retried create mints
// a second issue UUID and both eventually publish.)
if let Err(e) = release_result {
tracing::error!(repo = %record.name, issue = %issue_id, err = %e,
"issue committed locally but durable upload failed — storage re-syncs on next upload");
if state
.repo_store
.pending_marker_exists(&record.owner_did, &record.name)
{
tracing::error!(repo = %record.name, issue = %issue_id, err = %e,
"issue committed locally but durable upload failed — storage re-syncs on next upload");
} else {
tracing::error!(repo = %record.name, issue = %issue_id, err = %e,
"issue committed locally with NO durable protection — failing the request");
return Err(AppError::Git(format!(
"issue stored locally but durability could not be guaranteed: {e}"
)));
}
}

// Bump trust score for the issue author — increment current score by 0.05
Expand Down Expand Up @@ -280,11 +293,22 @@ pub async fn close_issue(
let updated = close_result
.map_err(|e| AppError::Git(e.to_string()))?
.ok_or_else(|| AppError::RepoNotFound(format!("issue {issue_id} not found")))?;
// Committed locally + marker-protected: a durable-upload failure is
// recoverable, not a request failure (see create_issue).
// Recoverable only while the marker protects the committed mutation
// (see create_issue).
if let Err(e) = release_result {
tracing::error!(repo = %repo, issue = %issue_id, err = %e,
"issue close committed locally but durable upload failed — storage re-syncs on next upload");
if state
.repo_store
.pending_marker_exists(&record.owner_did, &record.name)
{
tracing::error!(repo = %repo, issue = %issue_id, err = %e,
"issue close committed locally but durable upload failed — storage re-syncs on next upload");
} else {
tracing::error!(repo = %repo, issue = %issue_id, err = %e,
"issue close committed locally with NO durable protection — failing the request");
return Err(AppError::Git(format!(
"issue close stored locally but durability could not be guaranteed: {e}"
)));
}
}

let issue: serde_json::Value = serde_json::from_str(&updated)
Expand Down
27 changes: 19 additions & 8 deletions crates/gitlawb-node/src/api/pulls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -228,15 +228,26 @@ pub async fn merge_pr(
let release_result = guard.release(merge_result.is_ok()).await;

let merge_sha = merge_result.map_err(|e| AppError::Git(e.to_string()))?;
// The merge commit exists locally and is protected by the pending-upload
// marker, so a durable-upload failure is recoverable — NOT a request
// failure. Failing here would leave the target ref already merged while
// the PR row stays open, and a retry cannot un-merge it; proceed so the
// DB status matches the git state, and let the next successful upload
// re-sync storage.
// A durable-upload failure is recoverable ONLY while the pending-upload
// marker protects the merge commit; verify it exists before choosing to
// proceed (proceeding keeps the DB status consistent with the already
// merged ref, which a retry cannot un-merge). Without the marker nothing
// protects the merge from a stale-archive rollback — fail the request so
// the inconsistency is surfaced instead of silently losable.
if let Err(e) = release_result {
tracing::error!(repo = %record.name, pr = %pr.id, err = %e,
"merge committed locally but durable upload failed — storage re-syncs on next upload");
if state
.repo_store
.pending_marker_exists(&record.owner_did, &record.name)
{
tracing::error!(repo = %record.name, pr = %pr.id, err = %e,
"merge committed locally but durable upload failed — storage re-syncs on next upload");
} else {
tracing::error!(repo = %record.name, pr = %pr.id, err = %e,
"merge committed locally with NO durable protection — failing the request");
return Err(AppError::Git(format!(
"merge applied locally but durability could not be guaranteed: {e}"
)));
}
}

state.db.merge_pr(&pr.id, &merger_did).await?;
Expand Down
112 changes: 41 additions & 71 deletions crates/gitlawb-node/src/api/repos.rs
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,9 @@ pub async fn create_repo(
// insert arbitrates nothing; the cross-node safety property is that
// failure compensation below only ever touches state THIS attempt created
// (its own row by id, its own local dir) and never deletes a storage key.
// init() publishes under the per-repo advisory lock, so a push arriving
// through the just-visible row serializes behind publication and can
// never be destroyed by this compensation.
let disk_path = store::repo_disk_path(&state.config.repos_dir, &owner_did, &req.name);
let now = Utc::now();
let record = crate::db::RepoRecord {
Expand All @@ -231,10 +234,10 @@ pub async fn create_repo(
};
state.db.create_repo(&record).await?;

// Create the bare repo locally and upload the initial archive. On failure,
// compensate by removing our own just-inserted row (keyed by our id) so a
// retry starts clean; init() already removes its local dir on upload
// failure.
// Create the bare repo locally and publish the initial archive (under the
// advisory lock). On failure, compensate by removing our own just-inserted
// row (keyed by our id) so a retry starts clean; create_published removes
// its local dir itself.
if let Err(e) = state.repo_store.init(&owner_did, &req.name).await {
// `{:#}` walks the anyhow chain to the leaf cause; the other git
// handlers log their failures, this one didn't.
Expand Down Expand Up @@ -1643,10 +1646,10 @@ pub async fn fork_repo(
// anything. Within one node's database the row is the claim on (owner,
// name) — a concurrent same-name fork loses at the insert with nothing on
// disk or in storage. Across nodes (per-node Postgres) the insert
// arbitrates nothing; the cross-node safety property is that the failure
// compensation below only ever removes state THIS attempt created (its own
// row by id, its own clone dir) and never touches a storage archive that
// may belong to a racing winner's live repo.
// arbitrates nothing; the cross-node safety properties are that
// publication runs under the per-repo advisory lock (below) and that the
// failure compensation only ever removes state THIS attempt created (its
// own row by id, its own clone dir) — never a storage archive.
let now = Utc::now();
let record = crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
Expand All @@ -1663,71 +1666,38 @@ pub async fn fork_repo(
};
state.db.create_repo(&record).await?;

// Any failure from here must undo the claim: remove our own row (keyed by
// our generated id) and whatever `git clone` left at `disk_path` — clone
// can create the destination before failing, and an orphaned dir makes a
// retry fail on an existing destination.
//
// KNOWN COMPOUND-FAILURE WINDOW: the row makes the fork pushable before
// the clone+upload complete, and this flow holds no advisory lock across
// that phase. A push acked in the window, whose own upload also failed,
// lives only in this dir — undoing the claim then discards it. Requires
// fork-upload failure + an interleaved push + that push's upload failure;
// the follow-up is to hold the advisory lock across clone+publication.
let undo_claim = |state: AppState, record_id: String, disk_path: std::path::PathBuf| async move {
match tokio::fs::remove_dir_all(&disk_path).await {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => tracing::warn!(
path = %disk_path.display(), err = %e,
"failed to clean up local fork dir after fork error"
),
}
// Also drop the pending-upload marker a failed upload just wrote: left
// behind, it would read as divergence and wedge a same-name recreate.
crate::git::repo_store::clear_pending_upload(&disk_path);
if let Err(e) = state.db.delete_repo_by_id(&record_id).await {
tracing::warn!(record_id = %record_id, err = %e,
// The whole clone-and-publish lifecycle runs under the per-repo advisory
// lock inside `create_published`: pushes to the just-visible row serialize
// on the same lock, so none can execute (let alone be destroyed by the
// compensation below) until publication has succeeded or been unwound.
// On failure, compensation removes only what THIS attempt created: its
// clone dir (inside create_published) and its own row, keyed by our
// generated id.
let clone_result = state
.repo_store
.create_published(&forker_did, &fork_name, |dest| {
let output = std::process::Command::new("git")
.args([
"clone",
"--mirror",
source_path.to_str().unwrap_or(""),
dest.to_str().unwrap_or(""),
])
.output()
.map_err(|e| anyhow::anyhow!("git clone --mirror failed: {e}"))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
anyhow::bail!("git clone --mirror failed: {stderr}");
}
Ok(())
})
.await;
if let Err(e) = clone_result {
if let Err(db_err) = state.db.delete_repo_by_id(&record.id).await {
tracing::warn!(record_id = %record.id, err = %db_err,
"failed to remove fork row after fork error");
}
};

// Clone the source repo as a mirror
let output = match std::process::Command::new("git")
.args([
"clone",
"--mirror",
source_path.to_str().unwrap_or(""),
disk_path.to_str().unwrap_or(""),
])
.output()
{
Ok(output) => output,
Err(e) => {
undo_claim(state.clone(), record.id.clone(), disk_path).await;
return Err(AppError::Git(format!("git clone --mirror failed: {e}")));
}
};

if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
undo_claim(state.clone(), record.id.clone(), disk_path).await;
return Err(AppError::Git(format!(
"git clone --mirror failed: {stderr}"
)));
}

// Upload fork to storage — fail closed if the durable upload fails rather
// than reporting a fork that only exists on this node's local disk.
if let Err(e) = state
.repo_store
.release_after_write(&forker_did, &fork_name)
.await
{
undo_claim(state.clone(), record.id.clone(), disk_path).await;
return Err(AppError::Git(format!(
"fork created locally but durable upload failed: {e}"
)));
return Err(AppError::Git(format!("fork failed: {e}")));
}

// Persist the proof so the fork carries it when it propagates to peers.
Expand Down
4 changes: 3 additions & 1 deletion crates/gitlawb-node/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,9 @@ pub struct Config {
/// directory), or `ipfs` (Kubo MFS). Empty = auto-detect: `s3` when a bucket
/// is set, else `fs` when a storage dir is set, else local-only. `ipfs` is
/// never auto-selected (`GITLAWB_IPFS_API` alone keeps its pinning-only
/// meaning) — set this explicitly to `ipfs` to opt in.
/// meaning) — set this explicitly to `ipfs` to opt in, and note the ipfs
/// backend requires ALL nodes to share one Kubo instance (MFS is
/// daemon-local, not shared network storage).
#[arg(long, env = "GITLAWB_STORAGE_BACKEND", default_value = "")]
pub storage_backend: String,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Expand Down
Loading
Loading