Skip to content

[SPARK-9054] [SQL] Rename RowOrdering to InterpretedOrdering; use newOrdering in more places #7408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 8 commits into from
Closed

[SPARK-9054] [SQL] Rename RowOrdering to InterpretedOrdering; use newOrdering in more places #7408

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
473d814
[SPARK-9054] [SQL] Rename RowOrdering to InterpretedOrdering; use new…
JoshRosen Jul 15, 2015
9d9b092
Back out unrelated Exchange changes, which are slated for a separate …
JoshRosen Jul 15, 2015
5eb7222
Back out unintentional change.
JoshRosen Jul 15, 2015
845dd19
Merge remote-tracking branch 'origin/master' into SPARK-9054
JoshRosen Jul 15, 2015
9e2abbb
Add wrapper to allow RangePartitioner's Ordering to be serialized
JoshRosen Jul 15, 2015
8060594
Merge remote-tracking branch 'josh/SPARK-9054' into SPARK-9054
JoshRosen Jul 15, 2015
8fb32c5
Merge remote-tracking branch 'origin/master' into SPARK-9054
JoshRosen Jul 17, 2015
15b7d92
Add wrapper to fix issue in TakeOrderedAndProject
JoshRosen Jul 17, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Back out unrelated Exchange changes, which are slated for a separate … 
…patch
  • Loading branch information
@JoshRosen
JoshRosen committed Jul 15, 2015
commit 9d9b092f01c573b73260c6acf69c3190d04f2c8c
8 changes: 4 additions & 4 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,16 +158,16 @@ case class Exchange(
val rdd = if (needToCopyObjectsBeforeShuffle(part, serializer)) {
child.execute().mapPartitions { iter =>
val hashExpressions = newMutableProjection(expressions, child.output)()
iter.map(r => (hashExpressions(r).hashCode, r.copy()))
iter.map(r => (hashExpressions(r).copy, r.copy()))
Copy link
Contributor

@rxin rxin Jul 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm since copy is a verb, it is better to keep the () i think. also we defined copy with ()

Copy link
Contributor Author

@JoshRosen JoshRosen Jul 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't mean to make this change in this PR, will back out this one line.

}
} else {
child.execute().mapPartitions { iter =>
val hashExpressions = newMutableProjection(expressions, child.output)()
val mutablePair = new MutablePair[Int, InternalRow]()
iter.map(r => mutablePair.update(hashExpressions(r).hashCode, r))
val mutablePair = new MutablePair[InternalRow, InternalRow]()
iter.map(r => mutablePair.update(hashExpressions(r), r))
}
}
val shuffled = new ShuffledRDD[Int, InternalRow, InternalRow](rdd, part)
val shuffled = new ShuffledRDD[InternalRow, InternalRow, InternalRow](rdd, part)
shuffled.setSerializer(serializer)
shuffled.map(_._2)

Expand Down