Skip to content

Commit 9daa246

Browse files
committed
replace chunks with chunk groups
chunks are hardcoded to 1KiB in Blake3. Chunk groups are an experimental feature of bao that we have implemented in bao-tree.
1 parent f66a50b commit 9daa246

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/app/docs/layers/blobs/page.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ All blobs within iroh are referred to by the BLAKE3 [1] hash of its content. BLA
1313

1414
We leverage the tree hash structure of BLAKE3 as a basis for incremental verification when data is sent over the network, as described by Section 6.4 “Verified Streaming” in [1] and implemented in [2]. Iroh caches all chunk hashes as external metadata, leaving the unaltered input blob as the canonical source of bytes. Verified streaming also facilitates _range requests:_ fetching a verifiable contiguous subsequence of a blob by streaming only the portions of the BLAKE3 binary tree required to verify the designated subsequence.
1515

16-
Chunk hashes are distinct from root hashes and only used during data transfer. The chunk size of BLAKE3 is a tunable constant that defaults to 1KiB, which results in a 6% overhead on on the size of the input blob. Increasing the chunk size reduces overhead, at the cost of requiring more data transfer before an incremental verification checkpoint is reached. The chunk size constant can be modified & recalculated *without affecting the root hash.* This opens the door to experiment with different chunk size constants, even at runtime. We intend to investigate chunk size optimization in future work.
16+
Chunk hashes are distinct from root hashes and only used during data transfer. The chunk group size of BAO is a tunable constant that defaults to 1KiB, which results in a 6% overhead on on the size of the input blob. Increasing the chunk size reduces overhead, at the cost of requiring more data transfer before an incremental verification checkpoint is reached. The chunk group size constant can be modified & recalculated *without affecting the root hash.* This opens the door to experiment with different chunk group size constants, even at runtime. We intend to investigate chunk size optimization in future work.
1717

1818
Root hashes are expressed as a Content identifier (CID) as defined in [3], making iroh an *IPFS system* capable of interoperating with other systems that use CIDs. In contrast to other IPFS systems, only root hashes are valid content identifiers, which enforces a strict 1-1 relationship between a Content Identifier and blob. This 1-1 relationship brings iroh into alignment with common whole-file checksum systems. A naive implementation of iroh can skip verified streaming entirely and use the the CID as a whole-file checksum.
1919

0 commit comments

Comments
 (0)