Skip to content

[SQL] [SPARK-6620] Speed up toDF() and rdd() functions by constructing converters in ScalaReflection #5279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vlyubin wants to merge 9 commits into from
Closed

[SQL] [SPARK-6620] Speed up toDF() and rdd() functions by constructing converters in ScalaReflection #5279

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
41b2aa9
Creating converters for ScalaReflection stuff, and more
vlyubin Mar 26, 2015
8cad6e2
Addressed review commments
vlyubin Mar 31, 2015
881dc60
Moved to a separate module; addressed review comments; one extra plac…
vlyubin Apr 8, 2015
afa3aa5
Minor refactoring, added license, removed debug output
vlyubin Apr 8, 2015
74301fa
Addressed review comments
vlyubin Apr 8, 2015
dec6802
Addresed review feedback
vlyubin Apr 8, 2015
c327bc9
Moved the only remaining function from DataTypeConversions to DateUtils
vlyubin Apr 8, 2015
11a20ec
Avoid creating a tuple
vlyubin Apr 10, 2015
e75a387
Changes to ScalaUDF
vlyubin Apr 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Avoid creating a tuple
  • Loading branch information
@vlyubin
vlyubin committed Apr 10, 2015
commit 11a20ec2285342247c0675411f0ebf4aadc6eb51
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ object CatalystTypeConverters {
val convertedMap: HashMap[Any, Any] = HashMap()
while (iter.hasNext) {
val entry = iter.next()
convertedMap += Tuple2(keyConverter(entry.getKey), valueConverter(entry.getValue))
convertedMap(keyConverter(entry.getKey)) = valueConverter(entry.getValue)
}
convertedMap

Expand Down