Skip to content

[SPARK-19852][PYSPARK][ML] Update Python API setHandleInvalid for StringIndexer #17237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VinceShieh wants to merge 5 commits into from
Closed

[SPARK-19852][PYSPARK][ML] Update Python API setHandleInvalid for StringIndexer #17237

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d94dc68
[SPARK-19852][PYSPARK][ML] Update Python API for StringIndexer setHan…
Mar 10, 2017
f1d9bcb
fix compilation issues
Mar 10, 2017
affeeb7
doctest
Mar 10, 2017
b4bb765
update doctest
Mar 10, 2017
1d2f28f
include changes made by SPARK-11569
Mar 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
include changes made by SPARK-11569 
Signed-off-by: VinceShieh <[email protected]>
  • Loading branch information
VinceShieh committed Mar 17, 2017
commit 1d2f28f2449691fe7efe3e7dfbfe577ad8a56c1f
10 changes: 5 additions & 5 deletions python/pyspark/ml/feature.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1936,7 +1936,7 @@ class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, JavaMLReadable, Ja
... key=lambda x: x[0])
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'a'), (4, 'a'), (5, 'c')]
>>> testData2 = sc.parallelize([Row(id=0, label="a"), Row(id=1, label="d"),
... Row(id=2, label="e")], 2)
... Row(id=2, label=None)], 2)
>>> dfKeep= spark.createDataFrame(testData2)
>>> modelKeep = stringIndexer.setHandleInvalid("keep").fit(stringIndDf)
>>> tdK = modelKeep.transform(dfKeep)
Expand All @@ -1962,10 +1962,10 @@ class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, JavaMLReadable, Ja
.. versionadded:: 1.4.0
"""

handleInvalid = Param(Params._dummy(), "handleInvalid", "how to handle unseen labels. " +
"Options are 'skip' (filter out rows with unseen labels), " +
"error (throw an error), or 'keep' (put unseen labels in a special " +
"additional bucket, at index numLabels).",
handleInvalid = Param(Params._dummy(), "handleInvalid", "how to handle invalid data (unseen " +
"labels or NULL values). Options are 'skip' (filter out rows with " +
"invalid data), error (throw an error), or 'keep' (put invalid data " +
"in a special additional bucket, at index numLabels).",
typeConverter=TypeConverters.toString)

@keyword_only
Expand Down