[SPARK-3961] [MLlib] [PySpark] Python API for mllib.feature

[davies]

Added completed Python API for MLlib.feature

Normalizer
StandardScalerModel
StandardScaler
HashTF
IDFModel
IDF

cc @mengxr

[SparkQA]

QA tests have started for PR 2819 at commit 8a50584.

• This patch merges cleanly.

[Ishiihara]

In Scala/Java Word2Vec implementation , we used setters to set parameters, should we keep the same interface at python side?

[davies]

It's good to have same interface crossing languages, but sometimes it looks wired to having the API that is designed for Java.

I'd like to simply the Python API a little bit (without introducing confusing), then Python programmer can feel better (in this case). We can find several similar cases in APIs of pyspark.RDD.

Does it make sense?

[SparkQA]

QA tests have started for PR 2819 at commit 486795f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2819 at commit 8a50584.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21784/
Test FAILed.

[SparkQA]

QA tests have finished for PR 2819 at commit 486795f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21785/
Test FAILed.

[SparkQA]

QA tests have started for PR 2819 at commit 7a1891a.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2819 at commit 7a1891a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21789/
Test FAILed.

[SparkQA]

QA tests have started for PR 2819 at commit a405ae7.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2819 at commit a405ae7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21790/
Test PASSed.

[SparkQA]

QA tests have started for PR 2819 at commit 59781b9.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2819 at commit 59781b9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21850/
Test PASSed.

[jkbradley]

could remove "(_)"

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22308/
Test PASSed.

[jkbradley]

@davies LGTM

@davies @Ishiihara This debate about whether the Python API should be Pythonic or match the Scala/Java API is tough. @mateiz has recommended the latter (match Scala/Java); it would certainly be good to converge on a community standard!

[mateiz]

So regarding the interface, as I mentioned to Joseph, I would like the interfaces to be the same so that people can easily copy code between the languages. Many people will see a Spark example in one language on a slide, and then try to do the same thing in their own program, so we want what to be super simple. So don't remove the getters and setters. In this particular case, it may be okay to support keyword args in addition to the getters / setters, since it will be obvious that there's another way to do that. But we should only do this if we're absolutely certain that these methods will have no required args in the future, because otherwise default and named arguments can mess things out.

[mateiz]

BTW we can also leave out the default args for now and add them later, if we want to take more time to decide this. But the Python API should definitely include all the methods in the Scala / Java one.

[davies]

@mateiz @jkbradley @Ishiihara I had revert the API changes in Word2Vec, also remove the keyword arguments.

[SparkQA]

Test build #22316 has started for PR 2819 at commit b628693.

• This patch merges cleanly.

[SparkQA]

Test build #22316 has finished for PR 2819 at commit b628693.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22316/
Test PASSed.

[mengxr]

Vector is not used

[SparkQA]

Test build #22346 has started for PR 2819 at commit 4f48f48.

• This patch merges cleanly.

[SparkQA]

Test build #22346 has finished for PR 2819 at commit 4f48f48.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class VectorTransformer(object):
  • class Normalizer(VectorTransformer):
  • class JavaModelWrapper(VectorTransformer):
  • class StandardScalerModel(JavaModelWrapper):
  • class StandardScaler(object):
  • class HashingTF(object):
  • class IDFModel(JavaModelWrapper):
  • class IDF(object):
  • class Word2VecModel(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22346/
Test PASSed.

[mengxr]

LGTM. Merged into master. Thanks!