Skip to content

Add a config to force using string view in benchmark #11514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 8 commits into from
Jul 19, 2024
Merged

Add a config to force using string view in benchmark #11514

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
205e9f2
add a knob to force string view in benchmark
XiangpengHao Jul 15, 2024
7326f1f
fix sql logic test
XiangpengHao Jul 15, 2024
c066d9b
update doc
XiangpengHao Jul 15, 2024
aeeb77e
fix ci
XiangpengHao Jul 17, 2024
9e2476a
fix ci only test
XiangpengHao Jul 17, 2024
9c11a27
Update benchmarks/src/util/options.rs
XiangpengHao Jul 18, 2024
f980fc6
Update datafusion/common/src/config.rs
XiangpengHao Jul 18, 2024
38633d2
update tests
alamb Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update doc
  • Loading branch information
@XiangpengHao
XiangpengHao committed Jul 17, 2024
commit c066d9bc68598d7ab9d0d3861083adf829718613
3 changes: 2 additions & 1 deletion docs/source/user-guide/configs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ Environment variables are read during `SessionConfig` initialisation so they mus
| datafusion.execution.listing_table_ignore_subdirectory | true | Should sub directories be ignored when scanning directories for data files. Defaults to true (ignores subdirectories), consistent with Hive. Note that this setting does not affect reading partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`). |
| datafusion.execution.enable_recursive_ctes | true | Should DataFusion support recursive CTEs |
| datafusion.execution.split_file_groups_by_statistics | false | Attempt to eliminate sorts by packing & sorting files with non-overlapping statistics into the same file groups. Currently experimental |
| datafusion.execution.keep_partition_by_columns | false | Should DataFusion keep the columns used for partition_by in the output RecordBatches |
| datafusion.execution.keep_partition_by_columns | false | Should Datafusion keep the columns used for partition_by in the output RecordBatches |
| datafusion.execution.schema_force_string_view | false | If true, the parquet reader will replace `Utf8/Utf8Large` with `Utf8View`, and `Binary/BinaryLarge` with `BinaryView`. |
| datafusion.optimizer.enable_distinct_aggregation_soft_limit | true | When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read. |
| datafusion.optimizer.enable_round_robin_repartition | true | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores |
| datafusion.optimizer.enable_topk_aggregation | true | When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible |
Expand Down