Skip to content

[SPARK-16525][SQL] Enable Row Based HashMap in HashAggregateExec #14176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ooq wants to merge 9 commits into from
Closed

[SPARK-16525][SQL] Enable Row Based HashMap in HashAggregateExec #14176

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4a5b81f
Enable row-based fast hashmap and set it as default in auto mode.
ooq Jul 27, 2016
7194394
Remove the configuration flag that sets maximum column width for vect…
ooq Jul 27, 2016
122cf18
Fix klen and vlen in FixedLengthRowBasedKeyValueBatch (making multipl…
ooq Jul 28, 2016
def94cc
Fix writing DecimalType bug in RowBasedHashMapGenerator.
ooq Jul 28, 2016
97bb7c1
Fix comments (unfinished)
ooq Aug 1, 2016
b9a4268
Fix comments (unfinished)
ooq Aug 1, 2016
e67ff5d
Add explicit tests for aggregate hash maps and other minor fixs.
ooq Aug 3, 2016
b32cb7b
Remove accidental/unnecessary import.
ooq Aug 3, 2016
a58314c
Update flag settings and tests.
ooq Aug 24, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions ...main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,10 +165,16 @@ private void freeCurrentPage() {
protected FixedLengthRowBasedKeyValueBatch(StructType keySchema, StructType valueSchema,
int maxRows, TaskMemoryManager manager) {
super(keySchema, valueSchema, maxRows, manager);
klen = keySchema.defaultSize()
+ UnsafeRow.calculateBitSetWidthInBytes(keySchema.length());
vlen = valueSchema.defaultSize()
+ UnsafeRow.calculateBitSetWidthInBytes(valueSchema.length());
int keySize = 0;
int valueSize = 0;
for (String name : keySchema.fieldNames()) {
keySize += (keySchema.apply(name).dataType().defaultSize() + 7) / 8 * 8;
Copy link
Member

@sameeragarwal sameeragarwal Aug 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a small comment about this implicit ceiling logic and the reason why schema.defaultSize() doesn't work.

}
for (String name : valueSchema.fieldNames()) {
valueSize += (valueSchema.apply(name).dataType().defaultSize() + 7) / 8 * 8;
}
klen = keySize + UnsafeRow.calculateBitSetWidthInBytes(keySchema.length());
vlen = valueSize + UnsafeRow.calculateBitSetWidthInBytes(valueSchema.length());
recordLength = klen + vlen + 8;
}
}
Loading