Skip to content

[SPARK-16525][SQL] Enable Row Based HashMap in HashAggregateExec #14176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ooq wants to merge 9 commits into from
Closed

[SPARK-16525][SQL] Enable Row Based HashMap in HashAggregateExec #14176

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4a5b81f
Enable row-based fast hashmap and set it as default in auto mode.
ooq Jul 27, 2016
7194394
Remove the configuration flag that sets maximum column width for vect…
ooq Jul 27, 2016
122cf18
Fix klen and vlen in FixedLengthRowBasedKeyValueBatch (making multipl…
ooq Jul 28, 2016
def94cc
Fix writing DecimalType bug in RowBasedHashMapGenerator.
ooq Jul 28, 2016
97bb7c1
Fix comments (unfinished)
ooq Aug 1, 2016
b9a4268
Fix comments (unfinished)
ooq Aug 1, 2016
e67ff5d
Add explicit tests for aggregate hash maps and other minor fixs.
ooq Aug 3, 2016
b32cb7b
Remove accidental/unnecessary import.
ooq Aug 3, 2016
a58314c
Update flag settings and tests.
ooq Aug 24, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove the configuration flag that sets maximum column width for vect… 
…orized hashmap
  • Loading branch information
@ooq
ooq committed Jul 27, 2016
commit 71943941ebe548e0a2c66d633893b7e2196b94a6
18 changes: 10 additions & 8 deletions sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,14 +499,16 @@ object SQLConf {
.intConf
.createWithDefault(40)

val VECTORIZED_AGG_MAP_MAX_COLUMNS =
SQLConfigBuilder("spark.sql.codegen.aggregate.map.columns.max")
val ENFORCE_FAST_AGG_MAP_IMPL =
Copy link
Member

@sameeragarwal sameeragarwal Aug 1, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe just call this FAST_AGG_MAP_IMPL?

SQLConfigBuilder("spark.sql.codegen.aggregate.map.enforce.impl")
Copy link
Member

@sameeragarwal sameeragarwal Aug 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also make sure that all references to the old config are also appropriately modified.

.internal()
.doc("Sets the maximum width of schema (aggregate keys + values) for which aggregate with" +
"keys uses an in-memory columnar map to speed up execution. Setting this to 0 effectively" +
"disables the columnar map")
.intConf
.createWithDefault(3)
.doc("Sets the implementation for fast hash map during aggregation. Could be one of the " +
"following: rowbased, vectorized, skip, auto. Defaults to auto, and should only be other " +
"values for testing purposes.")
.stringConf
.transform(_.toLowerCase())
.checkValues(Set("rowbased", "vectorized", "skip", "auto"))
.createWithDefault("auto")
Copy link
Member

@sameeragarwal sameeragarwal Aug 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this removes the entire vectorized hashmap code from the the test path even though we claim to support it. Let's make sure that we have explicit tests that test for both "rowbased" and "vectorized".


val FILE_SINK_LOG_DELETION = SQLConfigBuilder("spark.sql.streaming.fileSink.log.deletion")
.internal()
Expand Down Expand Up @@ -673,7 +675,7 @@ private[sql] class SQLConf extends Serializable with CatalystConf with Logging {

override def runSQLonFile: Boolean = getConf(RUN_SQL_ON_FILES)

def vectorizedAggregateMapMaxColumns: Int = getConf(VECTORIZED_AGG_MAP_MAX_COLUMNS)
def enforceFastAggHashMapImpl: String = getConf(ENFORCE_FAST_AGG_MAP_IMPL)

def variableSubstituteEnabled: Boolean = getConf(VARIABLE_SUBSTITUTE_ENABLED)

Expand Down