bench: add benchmark running script that recreates the published benchmarks (#267)

* feat: add benchmark running script that recreates the published benchmarks

* refactor: add error handling to run_command

Author: jacobtrombetta

benchmark/README.md

@@ -2,14 +2,17 @@
 
 ## Description
 
-Given the number of each commitment length, the total number of commitments, and the number of bytes used to represent data elements, we compute each commitment sequence, using for that curve25519 and ristretto group operations. The data elements are randomly generated (in a deterministic way, given that the same seed is always provided).
+Given the number of each commitment length, the total number of commitments, and the number of bytes used to represent data elements, we compute each commitment sequence, using for that Curve25519 and Ristretto group operations. The data elements are randomly generated (in a deterministic way, given that the same seed is always provided).
 
 ## Running the benchmarks
 
-To run the whole benchmark on the GPU (update accordingly to use CPU), execute:
+To run the whole benchmark suite from outside of the NIX development shell execute:
 
 ```
-docker run --rm -e TEST_TMPDIR=/root/.cache_bazel -v blitzar/benchmark/multi_commitment:/root/.cache_bazel -v "$PWD":/src -w /src --gpus all --privileged -it joestifler/blitzar:7.0 benchmark/multi_commitment/scripts/run_benchmark.py --backend gpu --output-dir benchmark/multi_commitment/.proof_results --force-rerun-bench 1 --run-bench-callgrind 1 --run-bench 1
+nix develop --command python3 ./benchmark/scripts/run_benchmarks.py <cpu|gpu>
 ```
 
-Some files are generated in this process. They can be found on `benchmark/multi_commitment/.proof_results/` directory.
+From inside the NIX development shell execute:
+```
+python3 ./benchmark/scripts/run_benchmarks.py <cpu|gpu>
+```

benchmark/scripts/run_benchmarks.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+
+import sys
+import subprocess
+import argparse
+
+
+def run_command(cmd):
+    """Run a shell command and print output"""
+    print(f"Running: {' '.join(cmd)}")
+    try:
+        subprocess.run(cmd, check=True)
+    except subprocess.CalledProcessError as e:
+        print(f"Error: Command '{' '.join(cmd)}' failed with return code {e.returncode}")
+        print(f"Output: {e.output.decode('utf-8') if e.output else 'No output'}")
+        sys.exit(1)
+
+def main():
+    # Set up argument parsing
+    parser = argparse.ArgumentParser(description='Run Blitzar benchmarks')
+    parser.add_argument('device', choices=['cpu', 'gpu'], help='Device to run benchmarks on')
+    args = parser.parse_args()
+
+    # Define benchmark parameters
+    sizes = [10000, 100000, 1000000]
+    num_samples = 10
+    num_commitments = [1, 10]
+    element_nbytes = [1, 32]
+    verbose = 0
+
+    # Multi commitment benchmark
+    for commitment in num_commitments:
+        for nbytes in element_nbytes:
+            for size in sizes:
+                cmd = [
+                    "bazel", "run", "-c", "opt", "//benchmark/multi_commitment:benchmark", "--",
+                    args.device, str(size), str(num_samples), str(commitment), str(nbytes), str(verbose)
+                ]
+                run_command(cmd)
+
+    # Inner product proof benchmark
+    for size in sizes:
+        cmd = [
+            "bazel", "run", "-c", "opt", "//benchmark/inner_product_proof:benchmark", "--",
+            args.device, str(size), str(num_samples)
+        ]
+        run_command(cmd)
+
+
+if __name__ == "__main__":
+    main()