Skip to content

[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations #5725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 59 commits into from
Closed

[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations #5725

Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
480a74a
Initial import of code from Databricks unsafe utils repo.
JoshRosen Apr 17, 2015
ab68e08
Begin merging the UTF8String implementations.
JoshRosen Apr 18, 2015
f03e9c1
Play around with Unsafe implementations of more string methods.
JoshRosen Apr 18, 2015
5d55cef
Add skeleton for Row implementation.
JoshRosen Apr 18, 2015
8a8f9df
Add skeleton for GeneratedAggregate integration.
JoshRosen Apr 18, 2015
1ff814d
Add reminder to free memory on iterator completion
JoshRosen Apr 18, 2015
53ba9b7
Start prototyping Java Row -> UnsafeRow converters
JoshRosen Apr 19, 2015
fc4c3a8
Sketch how the converters will be used in UnsafeGeneratedAggregate
JoshRosen Apr 19, 2015
1a483c5
First version that passes some aggregation tests:
JoshRosen Apr 19, 2015
079f1bf
Some clarification of the BytesToBytesMap.lookup() / set() contract.
JoshRosen Apr 19, 2015
f764d13
Simplify address + length calculation in Location.
JoshRosen Apr 19, 2015
c754ae1
Now that the store*() contract has been stregthened, we can remove an…
JoshRosen Apr 19, 2015
ae39694
Add finalizer as "cleanup method of last resort"
JoshRosen Apr 19, 2015
c7f0b56
Reuse UnsafeRow pointer in UnsafeRowConverter
JoshRosen Apr 20, 2015
62ab054
Optimize for fact that get() is only called on String columns.
JoshRosen Apr 20, 2015
c55bf66
Free buffer once iterator has been fully consumed.
JoshRosen Apr 20, 2015
738fa33
Add feature flag to guard UnsafeGeneratedAggregate
JoshRosen Apr 20, 2015
c1b3813
Fix bug in UnsafeMemoryAllocator.free():
JoshRosen Apr 20, 2015
7df6008
Optimizations related to zeroing out memory:
JoshRosen Apr 21, 2015
58ac393
Use UNSAFE allocator in GeneratedAggregate (TODO: make this configura…
JoshRosen Apr 21, 2015
d2bb986
Update to implement new Row methods added upstream
JoshRosen Apr 22, 2015
b3eaccd
Extract aggregation map into its own class.
JoshRosen Apr 22, 2015
bade966
Comment update (bumping to refresh GitHub cache...)
JoshRosen Apr 22, 2015
d85eeff
Add basic sanity test for UnsafeFixedWidthAggregationMap
JoshRosen Apr 22, 2015
1f4b716
Merge Unsafe code into the regular GeneratedAggregate, guarded by a
JoshRosen Apr 22, 2015
92d5a06
Address a number of minor code review comments.
JoshRosen Apr 23, 2015
628f936
Use ints intead of longs for indexing.
JoshRosen Apr 23, 2015
23a440a
Bump up default hash map size
JoshRosen Apr 23, 2015
765243d
Enable optional performance metrics for hash map.
JoshRosen Apr 23, 2015
b26f1d3
Fix bug in murmur hash implementation.
JoshRosen Apr 23, 2015
49aed30
More long -> int conversion.
JoshRosen Apr 23, 2015
29a7575
Remove debug logging
JoshRosen Apr 24, 2015
ef6b3d3
Fix a bunch of FindBugs and IntelliJ inspections
JoshRosen Apr 24, 2015
06e929d
More warning cleanup
JoshRosen Apr 24, 2015
854201a
Import and comment cleanup
JoshRosen Apr 24, 2015
f3dcbfe
More mod replacement
JoshRosen Apr 24, 2015
afe8dca
Some Javadoc cleanup
JoshRosen Apr 24, 2015
a95291e
Cleanups to string handling code
JoshRosen Apr 24, 2015
31eaabc
Lots of TODO and doc cleanup.
JoshRosen Apr 24, 2015
6ffdaa1
Null handling improvements in UnsafeRow.
JoshRosen Apr 24, 2015
9c19fc0
Add configuration options for heap vs. offheap
JoshRosen Apr 24, 2015
cde4132
Add missing pom.xml
JoshRosen Apr 26, 2015
0925847
Disable MiMa checks for new unsafe module
JoshRosen Apr 27, 2015
a8e4a3f
Introduce MemoryManager interface; add to SparkEnv.
JoshRosen Apr 28, 2015
b45f070
Don't redundantly store the offset from key to value, since we can co…
JoshRosen Apr 28, 2015
162caf7
Fix test compilation
JoshRosen Apr 28, 2015
3ca84b2
Only zero the used portion of groupingKeyConversionScratchSpace
JoshRosen Apr 28, 2015
529e571
Measure timeSpentResizing in nanoseconds instead of milliseconds.
JoshRosen Apr 28, 2015
ce3c565
More comments, formatting, and code cleanup.
JoshRosen Apr 28, 2015
78a5b84
Add logging to MemoryManager
JoshRosen Apr 28, 2015
a19e066
Rename unsafe Java test suites to match Scala test naming convention.
JoshRosen Apr 28, 2015
de5e001
Fix debug vs. trace in logging message.
JoshRosen Apr 28, 2015
6e4b192
Remove an unused method from ByteArrayMethods.
JoshRosen Apr 28, 2015
70a39e4
Split MemoryManager into ExecutorMemoryManager and TaskMemoryManager:
JoshRosen Apr 28, 2015
50e9671
Throw memory leak warning even in case of error; add warning about co…
JoshRosen Apr 29, 2015
017b2dc
Remove BytesToBytesMap.finalize()
JoshRosen Apr 29, 2015
1bc36cc
Refactor UnsafeRowConverter to avoid unnecessary boxing.
JoshRosen Apr 29, 2015
81f34f8
Follow 'place children last' convention for GeneratedAggregate
JoshRosen Apr 29, 2015
eeee512
Add converters for Null, Boolean, Byte, and Short columns.
JoshRosen Apr 29, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove debug logging
  • Loading branch information
@JoshRosen
JoshRosen committed Apr 24, 2015
commit 29a75754d46fe2e24ce0f8903e169b4b37beec7f
1 change: 0 additions & 1 deletion unsafe/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,7 +593,6 @@ private void growAndRehash() {
allocator.free(oldLongArray.memoryBlock());
allocator.free(oldBitSet.memoryBlock());
if (enablePerfMetrics) {
System.out.println("Resizing took " + (System.currentTimeMillis() - resizeStartTime) + " ms");
timeSpentResizingMs += System.currentTimeMillis() - resizeStartTime;
}
}
Expand Down