Skip to content

[SPARK-30414][SQL] ParquetRowConverter optimizations: arrays, maps, plus misc. constant factors #27089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 6 commits into from
Closed

[SPARK-30414][SQL] ParquetRowConverter optimizations: arrays, maps, plus misc. constant factors #27089

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
90cebf0
Add private[this] to vals
JoshRosen Jan 3, 2020
7318785
Save a .updater() call for each field
JoshRosen Jan 3, 2020
e05de15
Only call currentRow.numFields once per loop
JoshRosen Jan 3, 2020
c7d1534
Replace Scala ArrayBuffer with Java ArrayList; clear buffer and re-us…
JoshRosen Jan 3, 2020
4456f91
Avoid GenericArrayData constructor perf. problems (see SPARK-30413)
JoshRosen Jan 3, 2020
6d16f59
Roll back to using Scala ArrayBuffer, but continue using clear()
joshrosen-stripe Jan 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Only call currentRow.numFields once per loop
  • Loading branch information
@JoshRosen
JoshRosen committed Jan 3, 2020
commit e05de150813b639929c18af1df09ec718d2d16fc
3 changes: 2 additions & 1 deletion ...c/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,8 @@ private[parquet] class ParquetRowConverter(

override def start(): Unit = {
var i = 0
while (i < currentRow.numFields) {
val numFields = currentRow.numFields
while (i < numFields) {
currentRow.setNullAt(i)
i += 1
}
Expand Down