Prefetch Row Groups using `next_reader` API in parquet-rs #18391

alamb · 2025-10-30T19:56:21Z

Which issue does this PR close?

Part of [Parquet] Pre-fetch the next row group when reading parquet files #18470

Rationale for this change

While messing arund with the push decoder in

Rewrite ParquetRecordBatchStream in terms of the PushDecoder arrow-rs#8159

I stumbled upon the ParquetRecordBatchStream::next_row_group API added by @Xuanwo upstream in parquet:

feat(parquet): Add next_row_group API for ParquetRecordBatchStream arrow-rs#6907

This API allows starting the IO work for the next reader while processing the current reader which should
allow better overlapping of IO and CPU work.

I figured I would give it a quick test in DataFusion to see if it actually helps.

What changes are included in this PR?

Add a prefetch stream that uses ParquetRecordBatchStream::next_row_group

Note: this will buffer more memory (potentially has an extra row group fetched in practice)

Are these changes tested?

Are there any user-facing changes?

alamb · 2025-10-30T19:57:02Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (e773e91) to 607325a diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-10-30T20:09:51Z

datafusion/datasource-parquet/src/opener.rs

+{
+    pub fn new(stream: ParquetRecordBatchStream<T>) -> Self {
+        Self {
+            state: EagerPrefetchState::next_row_group(stream),


in real code, this shouldn't start until the first batch is requested I suspect

alamb · 2025-10-30T20:25:06Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.76 ms │                   2.81 ms │     no change │
│ QQuery 1     │    54.49 ms │                  51.45 ms │ +1.06x faster │
│ QQuery 2     │   136.74 ms │                 140.55 ms │     no change │
│ QQuery 3     │   163.25 ms │                 150.59 ms │ +1.08x faster │
│ QQuery 4     │  1057.28 ms │                1268.61 ms │  1.20x slower │
│ QQuery 5     │  1516.79 ms │                1703.05 ms │  1.12x slower │
│ QQuery 6     │     2.79 ms │                   2.29 ms │ +1.22x faster │
│ QQuery 7     │    73.91 ms │                  66.43 ms │ +1.11x faster │
│ QQuery 8     │  1438.49 ms │                1587.57 ms │  1.10x slower │
│ QQuery 9     │  1884.87 ms │                1909.48 ms │     no change │
│ QQuery 10    │   497.14 ms │                 505.08 ms │     no change │
│ QQuery 11    │   573.33 ms │                 556.83 ms │     no change │
│ QQuery 12    │  1574.61 ms │                1702.14 ms │  1.08x slower │
│ QQuery 13    │  2511.95 ms │                2621.81 ms │     no change │
│ QQuery 14    │  1692.76 ms │                1685.45 ms │     no change │
│ QQuery 15    │  1213.54 ms │                1355.25 ms │  1.12x slower │
│ QQuery 16    │  2642.39 ms │                2754.08 ms │     no change │
│ QQuery 17    │  2652.39 ms │                2732.06 ms │     no change │
│ QQuery 18    │  4880.10 ms │                5030.49 ms │     no change │
│ QQuery 19    │   150.98 ms │                 128.87 ms │ +1.17x faster │
│ QQuery 20    │  1991.06 ms │                1931.54 ms │     no change │
│ QQuery 21    │  2521.16 ms │                2317.17 ms │ +1.09x faster │
│ QQuery 22    │  4199.03 ms │                4004.83 ms │     no change │
│ QQuery 23    │  1159.64 ms │                1252.93 ms │  1.08x slower │
│ QQuery 24    │   263.44 ms │                 234.09 ms │ +1.13x faster │
│ QQuery 25    │   722.12 ms │                 619.72 ms │ +1.17x faster │
│ QQuery 26    │   366.48 ms │                 335.47 ms │ +1.09x faster │
│ QQuery 27    │  3182.51 ms │                3124.72 ms │     no change │
│ QQuery 28    │ 23668.27 ms │               23754.28 ms │     no change │
│ QQuery 29    │   975.13 ms │                 990.16 ms │     no change │
│ QQuery 30    │  2172.76 ms │                2147.25 ms │     no change │
│ QQuery 31    │  2089.65 ms │                2088.47 ms │     no change │
│ QQuery 32    │  4630.22 ms │                4337.73 ms │ +1.07x faster │
│ QQuery 33    │  5903.03 ms │                5754.42 ms │     no change │
│ QQuery 34    │  6612.56 ms │                6143.61 ms │ +1.08x faster │
│ QQuery 35    │  2123.00 ms │                2086.54 ms │     no change │
│ QQuery 36    │    26.49 ms │                  18.69 ms │ +1.42x faster │
│ QQuery 37    │    26.22 ms │                  18.02 ms │ +1.45x faster │
│ QQuery 38    │    26.26 ms │                  17.90 ms │ +1.47x faster │
│ QQuery 39    │    25.51 ms │                  18.54 ms │ +1.38x faster │
│ QQuery 40    │    26.89 ms │                  19.22 ms │ +1.40x faster │
│ QQuery 41    │    26.08 ms │                  18.05 ms │ +1.44x faster │
│ QQuery 42    │    25.48 ms │                  18.35 ms │ +1.39x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 87483.56ms │
│ Total Time (alamb_prefetch_row_groups)   │ 87206.58ms │
│ Average Time (HEAD)                      │  2034.50ms │
│ Average Time (alamb_prefetch_row_groups) │  2028.06ms │
│ Queries Faster                           │         18 │
│ Queries Slower                           │          6 │
│ Queries with No Change                   │         19 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘

alamb · 2025-10-30T20:25:09Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (e773e91) to 607325a diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-10-30T21:00:21Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  2823.45 ms │                2784.58 ms │ no change │
│ QQuery 1     │  1371.93 ms │                1322.79 ms │ no change │
│ QQuery 2     │  2534.89 ms │                2497.29 ms │ no change │
│ QQuery 3     │  1186.66 ms │                1179.14 ms │ no change │
│ QQuery 4     │  2245.30 ms │                2233.54 ms │ no change │
│ QQuery 5     │ 28054.14 ms │               28501.99 ms │ no change │
│ QQuery 6     │  4266.42 ms │                4209.75 ms │ no change │
│ QQuery 7     │  3486.96 ms │                3544.71 ms │ no change │
└──────────────┴─────────────┴───────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 45969.76ms │
│ Total Time (alamb_prefetch_row_groups)   │ 46273.79ms │
│ Average Time (HEAD)                      │  5746.22ms │
│ Average Time (alamb_prefetch_row_groups) │  5784.22ms │
│ Queries Faster                           │          0 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │          8 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.36 ms │                   2.27 ms │     no change │
│ QQuery 1     │    51.11 ms │                  48.95 ms │     no change │
│ QQuery 2     │   141.29 ms │                 135.05 ms │     no change │
│ QQuery 3     │   167.36 ms │                 154.86 ms │ +1.08x faster │
│ QQuery 4     │  1106.24 ms │                1057.82 ms │     no change │
│ QQuery 5     │  1559.60 ms │                1476.47 ms │ +1.06x faster │
│ QQuery 6     │     3.01 ms │                   2.19 ms │ +1.38x faster │
│ QQuery 7     │    59.08 ms │                  54.06 ms │ +1.09x faster │
│ QQuery 8     │  1500.60 ms │                1450.38 ms │     no change │
│ QQuery 9     │  1997.55 ms │                1858.25 ms │ +1.07x faster │
│ QQuery 10    │   403.92 ms │                 400.80 ms │     no change │
│ QQuery 11    │   462.15 ms │                 447.32 ms │     no change │
│ QQuery 12    │  1459.95 ms │                1419.69 ms │     no change │
│ QQuery 13    │  2244.78 ms │                2203.38 ms │     no change │
│ QQuery 14    │  1323.72 ms │                1275.75 ms │     no change │
│ QQuery 15    │  1239.85 ms │                1234.18 ms │     no change │
│ QQuery 16    │  2684.91 ms │                2690.00 ms │     no change │
│ QQuery 17    │  2707.13 ms │                2657.77 ms │     no change │
│ QQuery 18    │  5279.35 ms │                5321.36 ms │     no change │
│ QQuery 19    │   128.38 ms │                 131.67 ms │     no change │
│ QQuery 20    │  2064.79 ms │                2031.68 ms │     no change │
│ QQuery 21    │  2373.89 ms │                2304.80 ms │     no change │
│ QQuery 22    │  4088.61 ms │                3966.60 ms │     no change │
│ QQuery 23    │ 13086.42 ms │               12736.79 ms │     no change │
│ QQuery 24    │   233.64 ms │                 211.33 ms │ +1.11x faster │
│ QQuery 25    │   515.82 ms │                 493.73 ms │     no change │
│ QQuery 26    │   226.98 ms │                 212.15 ms │ +1.07x faster │
│ QQuery 27    │  2943.27 ms │                2840.49 ms │     no change │
│ QQuery 28    │ 22798.29 ms │               22635.28 ms │     no change │
│ QQuery 29    │   975.66 ms │                 974.08 ms │     no change │
│ QQuery 30    │  1364.53 ms │                1322.54 ms │     no change │
│ QQuery 31    │  1380.77 ms │                1340.24 ms │     no change │
│ QQuery 32    │  4490.85 ms │                4857.64 ms │  1.08x slower │
│ QQuery 33    │  6085.71 ms │                6030.40 ms │     no change │
│ QQuery 34    │  6528.18 ms │                6442.43 ms │     no change │
│ QQuery 35    │  2119.99 ms │                2051.75 ms │     no change │
│ QQuery 36    │   122.07 ms │                 117.19 ms │     no change │
│ QQuery 37    │    53.10 ms │                  52.29 ms │     no change │
│ QQuery 38    │   119.36 ms │                 115.93 ms │     no change │
│ QQuery 39    │   199.59 ms │                 190.85 ms │     no change │
│ QQuery 40    │    45.33 ms │                  42.53 ms │ +1.07x faster │
│ QQuery 41    │    37.85 ms │                  37.55 ms │     no change │
│ QQuery 42    │    32.68 ms │                  31.88 ms │     no change │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 96409.71ms │
│ Total Time (alamb_prefetch_row_groups)   │ 95062.36ms │
│ Average Time (HEAD)                      │  2242.09ms │
│ Average Time (alamb_prefetch_row_groups) │  2210.75ms │
│ Queries Faster                           │          8 │
│ Queries Slower                           │          1 │
│ Queries with No Change                   │         34 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_prefetch_row_groups ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 171.28 ms │                 170.51 ms │    no change │
│ QQuery 2     │  27.34 ms │                  27.30 ms │    no change │
│ QQuery 3     │  41.46 ms │                  41.03 ms │    no change │
│ QQuery 4     │  29.93 ms │                  29.34 ms │    no change │
│ QQuery 5     │  81.54 ms │                  78.95 ms │    no change │
│ QQuery 6     │  20.01 ms │                  20.51 ms │    no change │
│ QQuery 7     │ 217.92 ms │                 224.19 ms │    no change │
│ QQuery 8     │  32.15 ms │                  35.18 ms │ 1.09x slower │
│ QQuery 9     │ 104.83 ms │                 106.04 ms │    no change │
│ QQuery 10    │  63.13 ms │                  63.51 ms │    no change │
│ QQuery 11    │  17.58 ms │                  17.53 ms │    no change │
│ QQuery 12    │  52.78 ms │                  51.65 ms │    no change │
│ QQuery 13    │  48.59 ms │                  47.58 ms │    no change │
│ QQuery 14    │  13.84 ms │                  14.03 ms │    no change │
│ QQuery 15    │  25.34 ms │                  24.75 ms │    no change │
│ QQuery 16    │  25.40 ms │                  25.16 ms │    no change │
│ QQuery 17    │ 154.56 ms │                 154.10 ms │    no change │
│ QQuery 18    │ 335.90 ms │                 329.76 ms │    no change │
│ QQuery 19    │  37.40 ms │                  37.05 ms │    no change │
│ QQuery 20    │  51.13 ms │                  49.64 ms │    no change │
│ QQuery 21    │ 331.96 ms │                 319.17 ms │    no change │
│ QQuery 22    │  20.14 ms │                  20.81 ms │    no change │
└──────────────┴───────────┴───────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 1904.22ms │
│ Total Time (alamb_prefetch_row_groups)   │ 1887.79ms │
│ Average Time (HEAD)                      │   86.56ms │
│ Average Time (alamb_prefetch_row_groups) │   85.81ms │
│ Queries Faster                           │         0 │
│ Queries Slower                           │         1 │
│ Queries with No Change                   │        21 │
│ Queries with Failure                     │         0 │
└──────────────────────────────────────────┴───────────┘

alamb · 2025-10-31T11:05:21Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (e773e91) to 607325a diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-10-31T11:20:36Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.56 ms │                   2.25 ms │ +1.14x faster │
│ QQuery 1     │    54.52 ms │                  49.73 ms │ +1.10x faster │
│ QQuery 2     │   140.87 ms │                 138.97 ms │     no change │
│ QQuery 3     │   164.70 ms │                 156.78 ms │     no change │
│ QQuery 4     │  1090.26 ms │                1042.83 ms │     no change │
│ QQuery 5     │  1508.79 ms │                1451.72 ms │     no change │
│ QQuery 6     │     2.20 ms │                   2.26 ms │     no change │
│ QQuery 7     │    70.68 ms │                  64.57 ms │ +1.09x faster │
│ QQuery 8     │  1435.73 ms │                1416.63 ms │     no change │
│ QQuery 9     │  1897.93 ms │                1784.60 ms │ +1.06x faster │
│ QQuery 10    │   493.35 ms │                 482.62 ms │     no change │
│ QQuery 11    │   572.32 ms │                 558.31 ms │     no change │
│ QQuery 12    │  1569.61 ms │                1480.79 ms │ +1.06x faster │
│ QQuery 13    │  2546.88 ms │                2423.75 ms │     no change │
│ QQuery 14    │  1650.95 ms │                1592.18 ms │     no change │
│ QQuery 15    │  1206.82 ms │                1214.39 ms │     no change │
│ QQuery 16    │  2623.99 ms │                2624.57 ms │     no change │
│ QQuery 17    │  2617.58 ms │                2653.60 ms │     no change │
│ QQuery 18    │  5288.80 ms │                4862.26 ms │ +1.09x faster │
│ QQuery 19    │   143.90 ms │                 129.09 ms │ +1.11x faster │
│ QQuery 20    │  1998.93 ms │                1888.85 ms │ +1.06x faster │
│ QQuery 21    │  2428.91 ms │                2259.22 ms │ +1.08x faster │
│ QQuery 22    │  4195.69 ms │                3945.80 ms │ +1.06x faster │
│ QQuery 23    │  1185.22 ms │                1174.87 ms │     no change │
│ QQuery 24    │   262.00 ms │                 244.40 ms │ +1.07x faster │
│ QQuery 25    │   694.76 ms │                 615.37 ms │ +1.13x faster │
│ QQuery 26    │   346.17 ms │                 323.69 ms │ +1.07x faster │
│ QQuery 27    │  3188.86 ms │                3020.59 ms │ +1.06x faster │
│ QQuery 28    │ 24187.59 ms │               23344.83 ms │     no change │
│ QQuery 29    │  1005.41 ms │                 984.78 ms │     no change │
│ QQuery 30    │  2188.45 ms │                2065.42 ms │ +1.06x faster │
│ QQuery 31    │  2069.10 ms │                2048.44 ms │     no change │
│ QQuery 32    │  4824.69 ms │                4395.26 ms │ +1.10x faster │
│ QQuery 33    │  5942.11 ms │                5610.59 ms │ +1.06x faster │
│ QQuery 34    │  6033.64 ms │                5988.40 ms │     no change │
│ QQuery 35    │  2079.90 ms │                2005.17 ms │     no change │
│ QQuery 36    │    25.56 ms │                  18.06 ms │ +1.42x faster │
│ QQuery 37    │    25.48 ms │                  17.54 ms │ +1.45x faster │
│ QQuery 38    │    25.57 ms │                  17.53 ms │ +1.46x faster │
│ QQuery 39    │    25.13 ms │                  17.77 ms │ +1.41x faster │
│ QQuery 40    │    26.99 ms │                  18.78 ms │ +1.44x faster │
│ QQuery 41    │    25.15 ms │                  18.17 ms │ +1.38x faster │
│ QQuery 42    │    25.46 ms │                  18.24 ms │ +1.40x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 87893.21ms │
│ Total Time (alamb_prefetch_row_groups)   │ 84173.70ms │
│ Average Time (HEAD)                      │  2044.03ms │
│ Average Time (alamb_prefetch_row_groups) │  1957.53ms │
│ Queries Faster                           │         24 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │         19 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘

alamb · 2025-10-31T11:20:39Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (e773e91) to 607325a diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-10-31T11:45:30Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  2815.07 ms │                2699.72 ms │ no change │
│ QQuery 1     │  1346.17 ms │                1297.14 ms │ no change │
│ QQuery 2     │  2520.90 ms │                2506.18 ms │ no change │
│ QQuery 3     │  1182.16 ms │                1158.12 ms │ no change │
│ QQuery 4     │  2221.49 ms │                2181.00 ms │ no change │
│ QQuery 5     │ 28008.30 ms │               27863.09 ms │ no change │
│ QQuery 6     │  4216.45 ms │                4199.96 ms │ no change │
│ QQuery 7     │  3386.36 ms │                3415.65 ms │ no change │
└──────────────┴─────────────┴───────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 45696.89ms │
│ Total Time (alamb_prefetch_row_groups)   │ 45320.86ms │
│ Average Time (HEAD)                      │  5712.11ms │
│ Average Time (alamb_prefetch_row_groups) │  5665.11ms │
│ Queries Faster                           │          0 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │          8 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.72 ms │                   2.42 ms │ +1.12x faster │
│ QQuery 1     │    50.78 ms │                  49.66 ms │     no change │
│ QQuery 2     │   139.34 ms │                 138.37 ms │     no change │
│ QQuery 3     │   161.93 ms │                 159.52 ms │     no change │
│ QQuery 4     │  1121.27 ms │                1078.05 ms │     no change │
│ QQuery 5     │  1550.32 ms │                1505.49 ms │     no change │
│ QQuery 6     │     2.19 ms │                   2.13 ms │     no change │
│ QQuery 7     │    54.99 ms │                  54.84 ms │     no change │
│ QQuery 8     │  1472.34 ms │                1465.87 ms │     no change │
│ QQuery 9     │  1899.12 ms │                1836.18 ms │     no change │
│ QQuery 10    │   399.04 ms │                 390.73 ms │     no change │
│ QQuery 11    │   458.29 ms │                 431.21 ms │ +1.06x faster │
│ QQuery 12    │  1430.97 ms │                1385.64 ms │     no change │
│ QQuery 13    │  2225.72 ms │                2175.40 ms │     no change │
│ QQuery 14    │  1332.40 ms │                1274.84 ms │     no change │
│ QQuery 15    │  1219.37 ms │                1219.05 ms │     no change │
│ QQuery 16    │  2674.83 ms │                2630.83 ms │     no change │
│ QQuery 17    │  2675.25 ms │                2627.48 ms │     no change │
│ QQuery 18    │  4944.30 ms │                4900.53 ms │     no change │
│ QQuery 19    │   128.41 ms │                 118.56 ms │ +1.08x faster │
│ QQuery 20    │  1976.26 ms │                1932.37 ms │     no change │
│ QQuery 21    │  2312.85 ms │                2229.28 ms │     no change │
│ QQuery 22    │  3952.35 ms │                3888.29 ms │     no change │
│ QQuery 23    │ 12921.20 ms │               12603.65 ms │     no change │
│ QQuery 24    │   223.57 ms │                 217.05 ms │     no change │
│ QQuery 25    │   538.34 ms │                 496.36 ms │ +1.08x faster │
│ QQuery 26    │   217.52 ms │                 206.25 ms │ +1.05x faster │
│ QQuery 27    │  2877.62 ms │                2821.31 ms │     no change │
│ QQuery 28    │ 22809.30 ms │               22742.69 ms │     no change │
│ QQuery 29    │   993.56 ms │                 981.61 ms │     no change │
│ QQuery 30    │  1353.64 ms │                1306.34 ms │     no change │
│ QQuery 31    │  1368.91 ms │                1328.15 ms │     no change │
│ QQuery 32    │  4334.13 ms │                4285.84 ms │     no change │
│ QQuery 33    │  5831.41 ms │                5720.06 ms │     no change │
│ QQuery 34    │  6011.22 ms │                5855.95 ms │     no change │
│ QQuery 35    │  2043.61 ms │                2029.89 ms │     no change │
│ QQuery 36    │   120.19 ms │                 117.86 ms │     no change │
│ QQuery 37    │    51.83 ms │                  50.37 ms │     no change │
│ QQuery 38    │   121.53 ms │                 112.66 ms │ +1.08x faster │
│ QQuery 39    │   196.10 ms │                 188.77 ms │     no change │
│ QQuery 40    │    43.16 ms │                  38.74 ms │ +1.11x faster │
│ QQuery 41    │    38.36 ms │                  39.18 ms │     no change │
│ QQuery 42    │    33.10 ms │                  31.08 ms │ +1.07x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 94313.34ms │
│ Total Time (alamb_prefetch_row_groups)   │ 92670.56ms │
│ Average Time (HEAD)                      │  2193.33ms │
│ Average Time (alamb_prefetch_row_groups) │  2155.13ms │
│ Queries Faster                           │          8 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │         35 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 178.15 ms │                 169.62 ms │     no change │
│ QQuery 2     │  27.12 ms │                  26.06 ms │     no change │
│ QQuery 3     │  41.92 ms │                  41.59 ms │     no change │
│ QQuery 4     │  29.16 ms │                  29.18 ms │     no change │
│ QQuery 5     │  78.41 ms │                  77.17 ms │     no change │
│ QQuery 6     │  19.79 ms │                  19.83 ms │     no change │
│ QQuery 7     │ 213.77 ms │                 221.48 ms │     no change │
│ QQuery 8     │  34.71 ms │                  36.13 ms │     no change │
│ QQuery 9     │  99.66 ms │                 100.06 ms │     no change │
│ QQuery 10    │  61.65 ms │                  62.06 ms │     no change │
│ QQuery 11    │  17.36 ms │                  17.20 ms │     no change │
│ QQuery 12    │  51.44 ms │                  51.30 ms │     no change │
│ QQuery 13    │  47.61 ms │                  46.83 ms │     no change │
│ QQuery 14    │  14.62 ms │                  14.18 ms │     no change │
│ QQuery 15    │  24.86 ms │                  25.19 ms │     no change │
│ QQuery 16    │  24.85 ms │                  25.62 ms │     no change │
│ QQuery 17    │ 151.17 ms │                 145.87 ms │     no change │
│ QQuery 18    │ 328.76 ms │                 331.27 ms │     no change │
│ QQuery 19    │  36.56 ms │                  37.19 ms │     no change │
│ QQuery 20    │  49.11 ms │                  49.93 ms │     no change │
│ QQuery 21    │ 353.45 ms │                 331.67 ms │ +1.07x faster │
│ QQuery 22    │  20.64 ms │                  21.41 ms │     no change │
└──────────────┴───────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 1904.77ms │
│ Total Time (alamb_prefetch_row_groups)   │ 1880.82ms │
│ Average Time (HEAD)                      │   86.58ms │
│ Average Time (alamb_prefetch_row_groups) │   85.49ms │
│ Queries Faster                           │         1 │
│ Queries Slower                           │         0 │
│ Queries with No Change                   │        21 │
│ Queries with Failure                     │         0 │
└──────────────────────────────────────────┴───────────┘

alamb · 2025-10-31T14:32:43Z

The benchmark results actually look fairly promising. I'll try and confirm these results manually and then polish this PR up and mark it ready for review

Dandandan · 2025-10-31T14:39:37Z

Might as well run some tests agains s3 / object store to see if performance improvement is even better?

Dandandan · 2025-10-31T14:41:57Z

datafusion/datasource-parquet/src/opener.rs

+                                self.state = EagerPrefetchState::Done;
+                            } else {
+                                // immediately start reading the next row group
+                                self.state = EagerPrefetchState::next_row_group(stream);


Might even make this configurable (i.e. prefetch >=2 row groups?)

good call -- I will look into that. I am not sure buffering more than 1 row group will really help (if you have more than one reader buffered it means the rest of the plan can't consume the data fast enough (and we need to apply backpressure)

alamb · 2025-11-03T15:45:24Z

I filed #18391 to track this idea more

Xuanwo · 2025-11-03T16:39:47Z

Cool!

Partially written by Codex

alamb · 2025-11-21T21:03:21Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (01d01fb) to e6d1773 diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-11-21T21:32:43Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.19 ms │                   2.50 ms │  1.15x slower │
│ QQuery 1     │    53.34 ms │                  47.90 ms │ +1.11x faster │
│ QQuery 2     │   138.51 ms │                 133.74 ms │     no change │
│ QQuery 3     │   163.47 ms │                 156.04 ms │     no change │
│ QQuery 4     │  1091.27 ms │                1089.84 ms │     no change │
│ QQuery 5     │  1519.09 ms │                1479.78 ms │     no change │
│ QQuery 6     │     2.16 ms │                   2.13 ms │     no change │
│ QQuery 7     │    71.06 ms │                  62.34 ms │ +1.14x faster │
│ QQuery 8     │  1427.96 ms │                1450.66 ms │     no change │
│ QQuery 9     │  1925.78 ms │                1842.31 ms │     no change │
│ QQuery 10    │   499.03 ms │                 459.47 ms │ +1.09x faster │
│ QQuery 11    │   555.72 ms │                 515.48 ms │ +1.08x faster │
│ QQuery 12    │  1570.57 ms │                1466.22 ms │ +1.07x faster │
│ QQuery 13    │  2566.66 ms │                2416.41 ms │ +1.06x faster │
│ QQuery 14    │  1690.15 ms │                1577.75 ms │ +1.07x faster │
│ QQuery 15    │  1274.55 ms │                1251.44 ms │     no change │
│ QQuery 16    │  2671.88 ms │                2674.40 ms │     no change │
│ QQuery 17    │  2670.28 ms │                2666.56 ms │     no change │
│ QQuery 18    │  5453.22 ms │                4909.04 ms │ +1.11x faster │
│ QQuery 19    │   145.98 ms │                 131.02 ms │ +1.11x faster │
│ QQuery 20    │  2014.42 ms │                1811.79 ms │ +1.11x faster │
│ QQuery 21    │  2419.08 ms │                2297.57 ms │ +1.05x faster │
│ QQuery 22    │  4145.36 ms │                3984.56 ms │     no change │
│ QQuery 23    │  1160.70 ms │                1352.17 ms │  1.16x slower │
│ QQuery 24    │   248.62 ms │                 218.57 ms │ +1.14x faster │
│ QQuery 25    │   665.09 ms │                 609.75 ms │ +1.09x faster │
│ QQuery 26    │   347.78 ms │                 309.79 ms │ +1.12x faster │
│ QQuery 27    │  3115.09 ms │                2945.71 ms │ +1.06x faster │
│ QQuery 28    │ 24425.84 ms │               23710.05 ms │     no change │
│ QQuery 29    │  1007.31 ms │                 982.26 ms │     no change │
│ QQuery 30    │  2172.32 ms │                2017.39 ms │ +1.08x faster │
│ QQuery 31    │  2087.85 ms │                2096.86 ms │     no change │
│ QQuery 32    │  4963.01 ms │                4610.60 ms │ +1.08x faster │
│ QQuery 33    │  6089.44 ms │                5755.70 ms │ +1.06x faster │
│ QQuery 34    │  6029.42 ms │                5939.87 ms │     no change │
│ QQuery 35    │  1933.88 ms │                1878.56 ms │     no change │
│ QQuery 36    │    25.27 ms │                  18.48 ms │ +1.37x faster │
│ QQuery 37    │    25.26 ms │                  18.38 ms │ +1.37x faster │
│ QQuery 38    │    25.25 ms │                  17.76 ms │ +1.42x faster │
│ QQuery 39    │    25.50 ms │                  17.38 ms │ +1.47x faster │
│ QQuery 40    │    26.62 ms │                  18.27 ms │ +1.46x faster │
│ QQuery 41    │    25.27 ms │                  17.45 ms │ +1.45x faster │
│ QQuery 42    │    26.03 ms │                  17.66 ms │ +1.47x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 88497.28ms │
│ Total Time (alamb_prefetch_row_groups)   │ 84981.65ms │
│ Average Time (HEAD)                      │  2058.08ms │
│ Average Time (alamb_prefetch_row_groups) │  1976.32ms │
│ Queries Faster                           │         25 │
│ Queries Slower                           │          2 │
│ Queries with No Change                   │         16 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘

alamb · 2025-11-21T21:32:46Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (01d01fb) to e6d1773 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-11-21T22:16:46Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0     │  2586.57 ms │                2600.23 ms │    no change │
│ QQuery 1     │  1183.92 ms │                1313.49 ms │ 1.11x slower │
│ QQuery 2     │  2307.80 ms │                2448.24 ms │ 1.06x slower │
│ QQuery 3     │  1142.44 ms │                1171.26 ms │    no change │
│ QQuery 4     │  2314.46 ms │                2294.19 ms │    no change │
│ QQuery 5     │ 28193.48 ms │               28460.06 ms │    no change │
│ QQuery 6     │  4012.21 ms │                3995.12 ms │    no change │
│ QQuery 7     │  3506.43 ms │                3467.62 ms │    no change │
└──────────────┴─────────────┴───────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 45247.31ms │
│ Total Time (alamb_prefetch_row_groups)   │ 45750.22ms │
│ Average Time (HEAD)                      │  5655.91ms │
│ Average Time (alamb_prefetch_row_groups) │  5718.78ms │
│ Queries Faster                           │          0 │
│ Queries Slower                           │          2 │
│ Queries with No Change                   │          6 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.56 ms │                   2.25 ms │ +1.14x faster │
│ QQuery 1     │    51.47 ms │                  49.08 ms │     no change │
│ QQuery 2     │   136.49 ms │                 132.57 ms │     no change │
│ QQuery 3     │   167.11 ms │                 150.45 ms │ +1.11x faster │
│ QQuery 4     │  1168.06 ms │                1068.44 ms │ +1.09x faster │
│ QQuery 5     │  1563.44 ms │                1476.98 ms │ +1.06x faster │
│ QQuery 6     │     2.19 ms │                   2.14 ms │     no change │
│ QQuery 7     │    55.27 ms │                  52.86 ms │     no change │
│ QQuery 8     │  1528.15 ms │                1407.13 ms │ +1.09x faster │
│ QQuery 9     │  1950.83 ms │                1803.29 ms │ +1.08x faster │
│ QQuery 10    │   369.96 ms │                 366.87 ms │     no change │
│ QQuery 11    │   425.24 ms │                 423.94 ms │     no change │
│ QQuery 12    │  1464.47 ms │                1339.75 ms │ +1.09x faster │
│ QQuery 13    │  2223.61 ms │                2105.49 ms │ +1.06x faster │
│ QQuery 14    │  1353.29 ms │                1234.82 ms │ +1.10x faster │
│ QQuery 15    │  1306.64 ms │                1189.39 ms │ +1.10x faster │
│ QQuery 16    │  2727.27 ms │                2643.78 ms │     no change │
│ QQuery 17    │  2699.89 ms │                2636.76 ms │     no change │
│ QQuery 18    │  5046.54 ms │                4987.83 ms │     no change │
│ QQuery 19    │   130.41 ms │                 118.56 ms │ +1.10x faster │
│ QQuery 20    │  1988.25 ms │                1920.30 ms │     no change │
│ QQuery 21    │  2281.59 ms │                2285.87 ms │     no change │
│ QQuery 22    │  3885.57 ms │                3864.47 ms │     no change │
│ QQuery 23    │ 12918.38 ms │               12704.91 ms │     no change │
│ QQuery 24    │   218.13 ms │                 216.25 ms │     no change │
│ QQuery 25    │   473.92 ms │                 451.64 ms │     no change │
│ QQuery 26    │   230.63 ms │                 210.78 ms │ +1.09x faster │
│ QQuery 27    │  2793.53 ms │                2713.85 ms │     no change │
│ QQuery 28    │ 23791.66 ms │               23323.86 ms │     no change │
│ QQuery 29    │   969.69 ms │                 988.05 ms │     no change │
│ QQuery 30    │  1349.72 ms │                1283.49 ms │     no change │
│ QQuery 31    │  1417.99 ms │                1401.00 ms │     no change │
│ QQuery 32    │  5177.94 ms │                4902.76 ms │ +1.06x faster │
│ QQuery 33    │  5853.59 ms │                5768.50 ms │     no change │
│ QQuery 34    │  6080.22 ms │                5970.30 ms │     no change │
│ QQuery 35    │  1957.75 ms │                1837.77 ms │ +1.07x faster │
│ QQuery 36    │   121.42 ms │                 113.76 ms │ +1.07x faster │
│ QQuery 37    │    53.54 ms │                  51.68 ms │     no change │
│ QQuery 38    │   121.41 ms │                 115.47 ms │     no change │
│ QQuery 39    │   195.14 ms │                 184.44 ms │ +1.06x faster │
│ QQuery 40    │    43.28 ms │                  39.52 ms │ +1.10x faster │
│ QQuery 41    │    39.70 ms │                  35.94 ms │ +1.10x faster │
│ QQuery 42    │    31.71 ms │                  31.89 ms │     no change │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 96367.63ms │
│ Total Time (alamb_prefetch_row_groups)   │ 93608.88ms │
│ Average Time (HEAD)                      │  2241.11ms │
│ Average Time (alamb_prefetch_row_groups) │  2176.95ms │
│ Queries Faster                           │         18 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │         25 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_prefetch_row_groups ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 130.57 ms │                 136.96 ms │    no change │
│ QQuery 2     │  27.40 ms │                  29.34 ms │ 1.07x slower │
│ QQuery 3     │  39.36 ms │                  38.46 ms │    no change │
│ QQuery 4     │  29.51 ms │                  28.64 ms │    no change │
│ QQuery 5     │  89.60 ms │                  87.38 ms │    no change │
│ QQuery 6     │  20.09 ms │                  19.88 ms │    no change │
│ QQuery 7     │ 232.22 ms │                 229.99 ms │    no change │
│ QQuery 8     │  33.98 ms │                  35.22 ms │    no change │
│ QQuery 9     │  98.48 ms │                 111.35 ms │ 1.13x slower │
│ QQuery 10    │  66.66 ms │                  63.98 ms │    no change │
│ QQuery 11    │  18.84 ms │                  18.09 ms │    no change │
│ QQuery 12    │  51.28 ms │                  51.56 ms │    no change │
│ QQuery 13    │  47.16 ms │                  47.49 ms │    no change │
│ QQuery 14    │  13.91 ms │                  14.32 ms │    no change │
│ QQuery 15    │  25.27 ms │                  24.62 ms │    no change │
│ QQuery 16    │  25.57 ms │                  24.69 ms │    no change │
│ QQuery 17    │ 153.79 ms │                 152.28 ms │    no change │
│ QQuery 18    │ 277.72 ms │                 281.34 ms │    no change │
│ QQuery 19    │  37.45 ms │                  41.41 ms │ 1.11x slower │
│ QQuery 20    │  50.31 ms │                  50.34 ms │    no change │
│ QQuery 21    │ 325.46 ms │                 336.77 ms │    no change │
│ QQuery 22    │  18.26 ms │                  17.76 ms │    no change │
└──────────────┴───────────┴───────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 1812.90ms │
│ Total Time (alamb_prefetch_row_groups)   │ 1841.86ms │
│ Average Time (HEAD)                      │   82.40ms │
│ Average Time (alamb_prefetch_row_groups) │   83.72ms │
│ Queries Faster                           │         0 │
│ Queries Slower                           │         3 │
│ Queries with No Change                   │        19 │
│ Queries with Failure                     │         0 │
└──────────────────────────────────────────┴───────────┘

alamb · 2025-11-23T13:03:50Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (01d01fb) to e6d1773 diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-11-23T13:32:54Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.16 ms │                   2.55 ms │  1.18x slower │
│ QQuery 1     │    54.50 ms │                  47.16 ms │ +1.16x faster │
│ QQuery 2     │   136.70 ms │                 135.82 ms │     no change │
│ QQuery 3     │   187.99 ms │                 152.22 ms │ +1.24x faster │
│ QQuery 4     │  1105.21 ms │                1117.12 ms │     no change │
│ QQuery 5     │  1500.72 ms │                1489.27 ms │     no change │
│ QQuery 6     │     2.09 ms │                   2.19 ms │     no change │
│ QQuery 7     │    70.20 ms │                  60.31 ms │ +1.16x faster │
│ QQuery 8     │  1476.33 ms │                1478.49 ms │     no change │
│ QQuery 9     │  1882.03 ms │                1853.53 ms │     no change │
│ QQuery 10    │   496.37 ms │                 469.76 ms │ +1.06x faster │
│ QQuery 11    │   555.55 ms │                 540.34 ms │     no change │
│ QQuery 12    │  1583.97 ms │                1479.19 ms │ +1.07x faster │
│ QQuery 13    │  2621.98 ms │                2485.71 ms │ +1.05x faster │
│ QQuery 14    │  1703.67 ms │                1596.18 ms │ +1.07x faster │
│ QQuery 15    │  1316.16 ms │                1263.90 ms │     no change │
│ QQuery 16    │  2806.58 ms │                2712.98 ms │     no change │
│ QQuery 17    │  2788.03 ms │                2685.94 ms │     no change │
│ QQuery 18    │  5772.47 ms │                5000.84 ms │ +1.15x faster │
│ QQuery 19    │   144.14 ms │                 131.30 ms │ +1.10x faster │
│ QQuery 20    │  2051.83 ms │                1784.11 ms │ +1.15x faster │
│ QQuery 21    │  2460.77 ms │                2291.01 ms │ +1.07x faster │
│ QQuery 22    │  4261.73 ms │                4058.14 ms │     no change │
│ QQuery 23    │  1151.62 ms │                1317.19 ms │  1.14x slower │
│ QQuery 24    │   256.99 ms │                 217.07 ms │ +1.18x faster │
│ QQuery 25    │   663.29 ms │                 612.93 ms │ +1.08x faster │
│ QQuery 26    │   362.37 ms │                 309.74 ms │ +1.17x faster │
│ QQuery 27    │  3131.99 ms │                2947.84 ms │ +1.06x faster │
│ QQuery 28    │ 24611.30 ms │               23531.00 ms │     no change │
│ QQuery 29    │   999.24 ms │                 982.19 ms │     no change │
│ QQuery 30    │  2142.64 ms │                2044.83 ms │     no change │
│ QQuery 31    │  2104.36 ms │                2083.66 ms │     no change │
│ QQuery 32    │  5166.00 ms │                4708.60 ms │ +1.10x faster │
│ QQuery 33    │  6049.70 ms │                5874.02 ms │     no change │
│ QQuery 34    │  6273.11 ms │                6108.87 ms │     no change │
│ QQuery 35    │  1964.62 ms │                1925.42 ms │     no change │
│ QQuery 36    │    25.86 ms │                  18.53 ms │ +1.40x faster │
│ QQuery 37    │    25.40 ms │                  17.73 ms │ +1.43x faster │
│ QQuery 38    │    25.13 ms │                  17.99 ms │ +1.40x faster │
│ QQuery 39    │    25.30 ms │                  22.65 ms │ +1.12x faster │
│ QQuery 40    │    27.21 ms │                  17.94 ms │ +1.52x faster │
│ QQuery 41    │    25.50 ms │                  17.68 ms │ +1.44x faster │
│ QQuery 42    │    25.77 ms │                  18.31 ms │ +1.41x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 90038.60ms │
│ Total Time (alamb_prefetch_row_groups)   │ 85632.24ms │
│ Average Time (HEAD)                      │  2093.92ms │
│ Average Time (alamb_prefetch_row_groups) │  1991.45ms │
│ Queries Faster                           │         23 │
│ Queries Slower                           │          2 │
│ Queries with No Change                   │         18 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘

alamb · 2025-11-23T14:13:57Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/prefetch_row_groups (01d01fb) to e6d1773 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-11-23T14:57:59Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_prefetch_row_groups
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0     │  2605.40 ms │                2576.05 ms │    no change │
│ QQuery 1     │  1229.35 ms │                1298.13 ms │ 1.06x slower │
│ QQuery 2     │  2356.40 ms │                2475.20 ms │ 1.05x slower │
│ QQuery 3     │  1159.99 ms │                1134.05 ms │    no change │
│ QQuery 4     │  2278.07 ms │                2282.49 ms │    no change │
│ QQuery 5     │ 27907.35 ms │               28528.90 ms │    no change │
│ QQuery 6     │  4019.63 ms │                4035.18 ms │    no change │
│ QQuery 7     │  3387.47 ms │                3307.92 ms │    no change │
└──────────────┴─────────────┴───────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 44943.67ms │
│ Total Time (alamb_prefetch_row_groups)   │ 45637.92ms │
│ Average Time (HEAD)                      │  5617.96ms │
│ Average Time (alamb_prefetch_row_groups) │  5704.74ms │
│ Queries Faster                           │          0 │
│ Queries Slower                           │          2 │
│ Queries with No Change                   │          6 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.51 ms │                   2.14 ms │ +1.17x faster │
│ QQuery 1     │    50.54 ms │                  49.22 ms │     no change │
│ QQuery 2     │   137.42 ms │                 134.82 ms │     no change │
│ QQuery 3     │   162.19 ms │                 150.70 ms │ +1.08x faster │
│ QQuery 4     │  1140.90 ms │                1028.48 ms │ +1.11x faster │
│ QQuery 5     │  1549.85 ms │                1471.13 ms │ +1.05x faster │
│ QQuery 6     │     2.12 ms │                   2.06 ms │     no change │
│ QQuery 7     │    55.26 ms │                  52.06 ms │ +1.06x faster │
│ QQuery 8     │  1509.65 ms │                1385.42 ms │ +1.09x faster │
│ QQuery 9     │  1999.99 ms │                1815.99 ms │ +1.10x faster │
│ QQuery 10    │   378.84 ms │                 360.09 ms │     no change │
│ QQuery 11    │   429.70 ms │                 419.63 ms │     no change │
│ QQuery 12    │  1501.97 ms │                1299.49 ms │ +1.16x faster │
│ QQuery 13    │  2209.09 ms │                2068.73 ms │ +1.07x faster │
│ QQuery 14    │  1324.26 ms │                1254.43 ms │ +1.06x faster │
│ QQuery 15    │  1287.91 ms │                1191.81 ms │ +1.08x faster │
│ QQuery 16    │  2654.91 ms │                2610.28 ms │     no change │
│ QQuery 17    │  2643.54 ms │                2605.79 ms │     no change │
│ QQuery 18    │  4915.32 ms │                4775.52 ms │     no change │
│ QQuery 19    │   127.89 ms │                 114.09 ms │ +1.12x faster │
│ QQuery 20    │  2017.16 ms │                1932.73 ms │     no change │
│ QQuery 21    │  2299.88 ms │                2192.85 ms │     no change │
│ QQuery 22    │  3921.45 ms │                3846.70 ms │     no change │
│ QQuery 23    │ 12890.68 ms │               12729.17 ms │     no change │
│ QQuery 24    │   215.46 ms │                 204.23 ms │ +1.06x faster │
│ QQuery 25    │   483.38 ms │                 457.82 ms │ +1.06x faster │
│ QQuery 26    │   216.62 ms │                 205.15 ms │ +1.06x faster │
│ QQuery 27    │  2796.43 ms │                2704.04 ms │     no change │
│ QQuery 28    │ 23672.70 ms │               23226.71 ms │     no change │
│ QQuery 29    │   994.04 ms │                 977.80 ms │     no change │
│ QQuery 30    │  1345.01 ms │                1301.24 ms │     no change │
│ QQuery 31    │  1363.81 ms │                1380.33 ms │     no change │
│ QQuery 32    │  4824.19 ms │                4417.79 ms │ +1.09x faster │
│ QQuery 33    │  5652.73 ms │                5691.90 ms │     no change │
│ QQuery 34    │  5948.16 ms │                5829.17 ms │     no change │
│ QQuery 35    │  1916.99 ms │                1901.18 ms │     no change │
│ QQuery 36    │   121.24 ms │                 114.29 ms │ +1.06x faster │
│ QQuery 37    │    52.77 ms │                  53.22 ms │     no change │
│ QQuery 38    │   118.80 ms │                 116.01 ms │     no change │
│ QQuery 39    │   198.14 ms │                 183.99 ms │ +1.08x faster │
│ QQuery 40    │    43.27 ms │                  42.30 ms │     no change │
│ QQuery 41    │    39.79 ms │                  37.38 ms │ +1.06x faster │
│ QQuery 42    │    33.32 ms │                  31.14 ms │ +1.07x faster │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 95249.87ms │
│ Total Time (alamb_prefetch_row_groups)   │ 92369.04ms │
│ Average Time (HEAD)                      │  2215.11ms │
│ Average Time (alamb_prefetch_row_groups) │  2148.12ms │
│ Queries Faster                           │         20 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │         23 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_prefetch_row_groups ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 129.44 ms │                 136.96 ms │  1.06x slower │
│ QQuery 2     │  28.37 ms │                  29.11 ms │     no change │
│ QQuery 3     │  34.50 ms │                  38.43 ms │  1.11x slower │
│ QQuery 4     │  29.96 ms │                  29.26 ms │     no change │
│ QQuery 5     │  89.15 ms │                  88.25 ms │     no change │
│ QQuery 6     │  19.85 ms │                  19.64 ms │     no change │
│ QQuery 7     │ 222.65 ms │                 231.88 ms │     no change │
│ QQuery 8     │  33.74 ms │                  35.57 ms │  1.05x slower │
│ QQuery 9     │ 103.29 ms │                 107.94 ms │     no change │
│ QQuery 10    │  64.12 ms │                  64.05 ms │     no change │
│ QQuery 11    │  18.70 ms │                  17.67 ms │ +1.06x faster │
│ QQuery 12    │  53.06 ms │                  51.92 ms │     no change │
│ QQuery 13    │  51.27 ms │                  47.34 ms │ +1.08x faster │
│ QQuery 14    │  14.25 ms │                  14.03 ms │     no change │
│ QQuery 15    │  25.06 ms │                  25.38 ms │     no change │
│ QQuery 16    │  25.38 ms │                  25.34 ms │     no change │
│ QQuery 17    │ 155.72 ms │                 154.88 ms │     no change │
│ QQuery 18    │ 286.82 ms │                 283.69 ms │     no change │
│ QQuery 19    │  37.85 ms │                  38.68 ms │     no change │
│ QQuery 20    │  48.91 ms │                  49.59 ms │     no change │
│ QQuery 21    │ 323.01 ms │                 328.05 ms │     no change │
│ QQuery 22    │  18.21 ms │                  17.86 ms │     no change │
└──────────────┴───────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 1813.30ms │
│ Total Time (alamb_prefetch_row_groups)   │ 1835.53ms │
│ Average Time (HEAD)                      │   82.42ms │
│ Average Time (alamb_prefetch_row_groups) │   83.43ms │
│ Queries Faster                           │         2 │
│ Queries Slower                           │         3 │
│ Queries with No Change                   │        17 │
│ Queries with Failure                     │         0 │
└──────────────────────────────────────────┴───────────┘

github-actions bot added the datasource Changes to the datasource crate label Oct 30, 2025

alamb commented Oct 30, 2025

View reviewed changes

Dandandan reviewed Oct 31, 2025

View reviewed changes

alamb mentioned this pull request Nov 3, 2025

[Parquet] Pre-fetch the next row group when reading parquet files #18470

Open

alamb force-pushed the alamb/prefetch_row_groups branch from e773e91 to d791922 Compare November 20, 2025 17:58

github-actions bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate labels Nov 20, 2025

alamb force-pushed the alamb/prefetch_row_groups branch from 4f1deda to 9fcad0f Compare November 21, 2025 20:52

alamb changed the title ~~TEST prefetching Row Groups using next_reader API in parquet-rs~~ Prefetch Row Groups using next_reader API in parquet-rs Nov 21, 2025

alamb force-pushed the alamb/prefetch_row_groups branch from 9fcad0f to 992b9e7 Compare November 21, 2025 20:55

Prefetching Row Groups using next_reader API in parquet-rs

e4f3634

Partially written by Codex

alamb force-pushed the alamb/prefetch_row_groups branch from 992b9e7 to e4f3634 Compare November 21, 2025 21:00

clieanup

01d01fb

Cleanups

3a81112

Prefetch Row Groups using next_reader API in parquet-rs #18391

Are you sure you want to change the base?

Prefetch Row Groups using next_reader API in parquet-rs #18391

Uh oh!

Conversation

alamb commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

Dandandan commented Oct 31, 2025

Uh oh!

Dandandan Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 3, 2025

Uh oh!

Xuanwo commented Nov 3, 2025

Uh oh!

alamb commented Nov 21, 2025

Uh oh!

alamb commented Nov 21, 2025

Uh oh!

alamb commented Nov 21, 2025

Uh oh!

alamb commented Nov 21, 2025

Uh oh!

alamb commented Nov 23, 2025

Uh oh!

alamb commented Nov 23, 2025

Uh oh!

alamb commented Nov 23, 2025

Uh oh!

alamb commented Nov 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Prefetch Row Groups using `next_reader` API in parquet-rs #18391

Prefetch Row Groups using `next_reader` API in parquet-rs #18391

alamb commented Oct 30, 2025 •

edited

Loading