Skip to content

[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation #8038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 14 commits into from
Closed

[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation #8038

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
27f2e7f
Reserve memory in advance in TungstenAggregate
Aug 7, 2015
6549654
Actually request the memory in constructor + add tests
Aug 7, 2015
995be3d
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 7, 2015
ca1b44c
Minor: Update comment
Aug 7, 2015
355a9bd
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 7, 2015
2cf088a
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 10, 2015
4d416d0
Do not acquire a page if creating sorter destructively
Aug 10, 2015
70b8500
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 10, 2015
b4d3633
Address comments
Aug 10, 2015
b10a4f3
Fix test
Aug 11, 2015
94ca5de
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 11, 2015
d4dc9ca
Fix tests
Aug 12, 2015
19f2e1b
Fix tests again
Aug 12, 2015
7ebf6b9
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix test
  • Loading branch information
Andrew Or committed Aug 11, 2015
commit b10a4f3904bd1a67c0c5ae07e11e21464c6a0268
12 changes: 7 additions & 5 deletions ...est/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIteratorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,17 @@ package org.apache.spark.sql.execution.aggregate

import org.apache.spark._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.unsafe.memory.TaskMemoryManager
import org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection
import org.apache.spark.sql.test.TestSQLContext
import org.apache.spark.unsafe.memory.TaskMemoryManager

class TungstenAggregationIteratorSuite extends SparkFunSuite with LocalSparkContext {
class TungstenAggregationIteratorSuite extends SparkFunSuite {

test("memory acquired on construction") {
// Needed for various things in SparkEnv
sc = new SparkContext("local", "testing")
val taskMemoryManager = new TaskMemoryManager(sc.env.executorMemoryManager)
// set up environment
TestSQLContext

val taskMemoryManager = new TaskMemoryManager(SparkEnv.get.executorMemoryManager)
val taskContext = new TaskContextImpl(0, 0, 0, 0, taskMemoryManager, null, Seq.empty)
TaskContext.setTaskContext(taskContext)

Expand Down