[SPARK-10117][MLLIB] Implement SQL data source API for reading LIBSVM data

[Lewuathe]

It is convenient to implement data source API for LIBSVM format to have a better integration with DataFrames and ML pipeline API.

Two option is implemented.

• numFeatures: Specify the dimension of features vector
• featuresType: Specify the type of output vector. sparse is default.

[SparkQA]

Test build #41827 has finished for PR 8537 at commit 99accaa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class LibSVMRelation(val path: String, val numFeatures: Int, val featuresType: String)
  • class DefaultSource extends RelationProvider with DataSourceRegister
  • implicit class LibSVMReader(read: DataFrameReader)

[mengxr]

• featuresType -> vectorType?
• Do we need to make this class public? If yes, I would recommend a default public constructor with path and creating setter/getter for each option to make it easy to expand in the future.
• missing doc

[mengxr]

@Lewuathe Thanks for working on this! Beside inline comments, could you add a Java test suite?

[mengxr]

@Lewuathe Could you add [MLLIB] to the PR tile?

[Lewuathe]

@mengxr Thank you for reviewing and pointing! I'll update the patch and test codes.

[SparkQA]

Test build #41934 has finished for PR 8537 at commit 40d3027.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DefaultSource extends RelationProvider with DataSourceRegister
  • implicit class LibSVMReader(read: DataFrameReader)

[mengxr]

Since we need to read the entire file anyway, it doesn't save much with PrunedScan. Maybe TableScan is simpler but sufficient.

[mengxr]

@Lewuathe I made another pass.

[mengxr]

This doesn't verify the result is a sparse vector because runtime type erasure. We need

val v = row1.getAs[SparseVector](1)
assert(v == Vectors.sparse(...))

to force check.

[SparkQA]

Test build #42074 has finished for PR 8537 at commit 4f40891.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DefaultSource extends RelationProvider with DataSourceRegister
  • implicit class LibSVMReader(read: DataFrameReader)

[SparkQA]

Test build #42100 has finished for PR 8537 at commit 0ea1c1c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DefaultSource extends RelationProvider with DataSourceRegister

[Lewuathe]

Can't it find ml package in DataSourceRegister?

[rxin]

You need to add a file to the mllib module.

[SparkQA]

Test build #42129 has finished for PR 8537 at commit 9ce63c7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DefaultSource extends RelationProvider with DataSourceRegister

[mengxr]

Could you add @Since("1.6.0") to DefaultSource? See https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala#L128.

[mengxr]

Made another pass, only some minor issues left.

[mengxr]

LGTM pending Jenkins

[SparkQA]

Test build #42208 has finished for PR 8537 at commit 986999d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DefaultSource extends RelationProvider with DataSourceRegister

[mengxr]

Merged into master. Thanks!