Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

What changes were proposed in this pull request?

This PR aims to add a missing appendFloats API for array into ColumnVector class. For double type, there is appendDoubles for array here.

How was this patch tested?

Pass the Jenkins with a newly added test case.

…ats` for array

This PR aims to add a missing `appendFloats` API for array into **ColumnVector** class. For double type, there is `appendDoubles` for array [here](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java#L818-L824).

Pass the Jenkins with a newly added test case.

Author: Dongjoon Hyun <[email protected]>

Closes #17836 from dongjoon-hyun/SPARK-20566.
@SparkQA
Copy link

SparkQA commented Jul 7, 2017

Test build #79306 has finished for PR 18557 at commit 39839bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
This is the backport for #17836 .

@kiszk
Copy link
Member

kiszk commented Jul 7, 2017

@dongjoon-hyun Is there any reason to backport this to previous versions? This is because we had such a discussion.
Obviously, it makes sense to support the latest one.

@dongjoon-hyun
Copy link
Member Author

Hi, @kiszk .
I think this is a bug fix of ColumnVector as described in SPARK-20566.

@kiszk
Copy link
Member

kiszk commented Jul 7, 2017

We have not seen any failure in test suites. And, there is no usage of this API in Spark 2.2.

Does this missing cause any failure of test or application program? If so, it is good to put a sample program in this PR.

@cloud-fan
Copy link
Contributor

I've changed the ticket type from bug to improvement, adding a new API is not fixing a bug.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jul 7, 2017

I know that 'there is no usage of this API internally in Spark 2.2', but it's only for 2.2.0.
My reason was any 3rd party Spark library cannot use ColumnVector for float type in Spark 2.2.1+. It's performance issue.

Anyway, @cloud-fan changes the issue type. If that means backporting is not allowed for this patch, I have no objection for the community decision. I agree with that, too.

So, @kiszk and @cloud-fan . Given that, may I close this PR?

@dongjoon-hyun
Copy link
Member Author

BTW, thank you for swift reviews and feedbacks on my PR. :)

@cloud-fan
Copy link
Contributor

cloud-fan commented Jul 7, 2017

ColumnVector is totally internal in Spark 2.2, so there won't be 3rd party Spark library issue.

@dongjoon-hyun
Copy link
Member Author

Yep. It's totally internal officially.

What I meant with performance issue is 3rd party can still use it and there might be a performance gap between float and double.

I'll close this PR. Thank you again.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-20566-BRANCH-2.2 branch July 7, 2017 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants