UserOperation: Action-based Transactions for real diffs

[wjones127]

Create a new operation to improve transactions so that:

1. It's easy for users to read the transaction history
2. The operation is a true diff of the manifest

This is based on earlier proposal #3734. I've simplified the proposal, removing CompositeOperation and setting the advanced use cases aside.

Design

There will be two levels

User Operation

User operations are the user-facing ones. They will be the high-level operations that users recognize, such as INSERT, CREATE INDEX, etc.

message UserOperation {
  // Human-readable description, e.g. "INSERT INTO t VALUES (1);"
  string description = 1;
  // Unique identifier for this user operation
  UUID uuid = 2;
  // The version this operation was planned against
  uint64 read_version = 3;
  // The ordered list of manifest changes
  repeated Action actions = 4;
}

Actions

Action is a granular change to the manifest. Traditional operations will be made up of multiple of these.

message Action {
  oneof action {
    AddFragments add_fragments = 1;
    RemoveFragments remove_fragments = 2;
    UpdateFragment update_fragment = 3;
    AddFields add_fields = 4;
    DropFields drop_fields = 5;
    UpdateSchemaMetadata update_schema_metadata = 6;
    AddIndex add_index = 7;
    RemoveIndex remove_index = 8;
    UpdateConfig update_config = 9;
    UpdateTableMetadata update_table_metadata = 10;
    SetDataFormat set_data_format = 11;
    SetFeatureFlags set_feature_flags = 12;
  }
}

Details

// Append new fragments to the table.
// Covers: Append, the "add new rows" part of Overwrite, new fragments from compaction.
message AddFragments {
  repeated DataFragment fragments = 1;
}

// Remove fragments entirely by ID.
// Covers: the "remove old" part of Overwrite, removing compacted fragments.
message RemoveFragments {
  repeated uint64 fragment_ids = 1;
}

// Modify a single existing fragment. One message per fragment touched.
// Covers: Delete (new deletion file), Update (new data files + new deletion file),
//         Merge/AddColumns (new data files for new fields), DropColumns (tombstone fields).
message UpdateFragment {
  // The fragment being modified.
  uint64 fragment_id = 1;

  // --- Data file changes ---

  // New data files to add to this fragment (e.g. new column data, rewritten columns).
  repeated DataFile added_data_files = 2;
  // Data file paths to remove from this fragment (e.g. replaced by rewritten files).
  // Matched by DataFile.path.
  repeated string removed_data_file_paths = 3;

  // --- Deletion file changes ---

  // If set, replaces the fragment's current deletion file.
  // To clear an existing deletion file (e.g. after materializing deletions), set
  // clear_deletion_file = true instead.
  optional DeletionFile new_deletion_file = 4;
  // If true, remove the current deletion file (deletions have been materialized).
  bool clear_deletion_file = 5;

  // --- Row ID / lineage changes ---

  // Updated row ID sequence, if changed.
  oneof new_row_id_sequence {
    bytes inline_row_ids = 6;
    ExternalFile external_row_ids = 7;
  }

  // Updated row lineage tracking, if changed.
  oneof new_last_updated_at_version_sequence {
    bytes inline_last_updated_at_versions = 8;
    ExternalFile external_last_updated_at_versions = 9;
  }

  oneof new_created_at_version_sequence {
    bytes inline_created_at_versions = 10;
    ExternalFile external_created_at_versions = 11;
  }

  // Updated physical_rows, if it changed (e.g. after partial compaction within a fragment).
  optional uint64 physical_rows = 12;

  // Field IDs to tombstone (-2) in existing data files within this fragment.
  // Used for drop_column: marks fields as dead in their original data files without
  // removing the files (other fields in those files may still be live).
  repeated int32 tombstoned_field_ids = 13;
}

// ---- Schema Actions ----

// Add new fields to the schema.
// Covers: AddColumns / Merge, adding fields during Overwrite.
message AddFields {
  repeated lance.file.Field fields = 1;
  // Optional new entries for the schema metadata map.
  map<string, bytes> added_schema_metadata = 2;
}

// Drop fields from the schema by field ID.
// Covers: DropColumns.
// Note: The corresponding data is tombstoned via UpdateFragment.tombstoned_field_ids.
message DropFields {
  repeated int32 field_ids = 1;
  // Schema metadata keys to remove.
  repeated string removed_schema_metadata_keys = 2;
}

// Update schema-level metadata without changing fields.
message UpdateSchemaMetadata {
  map<string, bytes> set_values = 1;
  repeated string remove_keys = 2;
}

// ---- Index Actions ----

// Add or replace an index.
message AddIndex {
  // The full index metadata entry (same structure already used in the index section).
  IndexMetadata index = 1;
}

// Remove an index by its UUID.
message RemoveIndex {
  UUID index_uuid = 1;
}

// ---- Config / Metadata Actions ----

// Update table configuration (lance.* keys, etc.).
message UpdateConfig {
  map<string, string> set_values = 1;
  repeated string remove_keys = 2;
}

// Update table metadata (arbitrary user key-value pairs).
message UpdateTableMetadata {
  map<string, string> set_values = 1;
  repeated string remove_keys = 2;
}

// Set the data storage format.
message SetDataFormat {
  Manifest.DataStorageFormat format = 1;
}

// Set reader/writer feature flags. Values are OR'd onto existing flags.
message SetFeatureFlags {
  // Bits to set (OR) on reader_feature_flags.
  uint64 enable_reader_flags = 1;
  // Bits to set (OR) on writer_feature_flags.
  uint64 enable_writer_flags = 2;
}

Benefits

• Easier cleanup of cancelled transactions: can see what the set of new files in a transaction are more easily.
• Easier diffs between table versions
• Can do more complicated operations like append and update an index in a single transaction.

[westonpace]

Very cool. Should description be repeatable? Or maybe something like...

message UserOperation {
  // Unique identifier for this user operation
  UUID uuid = 2;
  // The version this operation was planned against
  uint64 read_version = 3;
  // The ordered list of user actions
  repeated UserAction actions = 4;
}

message UserAction {
  // Human-readable description, e.g. "INSERT INTO t VALUES (1);"
  string description = 1;
  // The ordered list of manifest changes
  repeated Action actions = 4;
}

I think a singular description might satisfy the "append and update in a single transaction" case but might not fully satisfy the branch-and-merge case. This would purely be for analyzing the transaction log. To apply the operation it should be fine to just flatten the list of actions.

[charleshuang119]

Love this design. The move from monolithic oneof Operation to composable repeated Action is a big improvement. Being able to expressAddFragments + AddIndex in one commit directly solves the performance cliff problem from #5952.

One gap I noticed: there's no tag-related action in the Action enum. In #5952 @beinan highlighted the need for atomic Data + Index + Tag commits, which is exactly our use case at Uber (we run on OCI-compatible objectstorage). Today, tags live as separate JSON files under _refs/tags/ via a plain PU, completely decoupled from the manifest commit.

Would it make sense to add:

message AddTag {
    string tag_name = 1;
}
message RemoveTag {
    string tag_name = 1;
}

as Action variants? That way, a UserOperation could bundle AddFragments + AddIndex + AddTag and the tag becomes visible atomically with the data and index. No second PUT, no split-brain window. (The existing Transaction proto already has a tag field, and Manifest has one too, but neither is connected to _refs/tags/. This new Action model seems like the right place to unify them.)

Also +1 to @westonpace's UserAction middle layer. For the branch-and-merge workflow in #5952, having per-step descriptions(e.g. "Worker A: append batch" / "Worker B: build index") would make the transaction log much easier to debug.