Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
e6e483c
[SPARK-9679] [ML] [PYSPARK] Add Python API for Stop Words Remover
holdenk Sep 1, 2015
3f63bd6
[SPARK-10398] [DOCS] Migrate Spark download page to use new lua mirro…
srowen Sep 1, 2015
ec01280
[SPARK-4223] [CORE] Support * in acls.
Sep 1, 2015
bf550a4
[SPARK-10162] [SQL] Fix the timezone omitting for PySpark Dataframe f…
0x0FFF Sep 1, 2015
00d9af5
[SPARK-10392] [SQL] Pyspark - Wrong DateType support on JDBC connection
0x0FFF Sep 1, 2015
c3b881a
[SPARK-7336] [HISTORYSERVER] Fix bug that applications status incorre…
ArcherShao Sep 2, 2015
56c4c17
[SPARK-10034] [SQL] add regression test for Sort on Aggregate
cloud-fan Sep 2, 2015
fc48307
[SPARK-10389] [SQL] support order by non-attribute grouping expressio…
cloud-fan Sep 2, 2015
2da3a9e
[SPARK-10004] [SHUFFLE] Perform auth checks when clients read shuffle…
Sep 2, 2015
6cd98c1
[SPARK-10417] [SQL] Iterating through Column results in infinite loop
0x0FFF Sep 2, 2015
03f3e91
[SPARK-10422] [SQL] String column in InMemoryColumnarCache needs to o…
yhuai Sep 3, 2015
44948a2
[SPARK-9723] [ML] params getordefault should throw more useful error
holdenk Sep 3, 2015
4bd85d0
[SPARK-5945] Spark should not retry a stage infinitely on a FetchFail…
Sep 3, 2015
0985d2c
[SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid pa…
navis Sep 3, 2015
f6c447f
Removed code duplication in ShuffleBlockFetcherIterator
eracah Sep 3, 2015
3ddb9b3
[SPARK-10247] [CORE] improve readability of a test case in DAGSchedul…
squito Sep 3, 2015
62b4690
[SPARK-10379] preserve first page in UnsafeShuffleExternalSorter
Sep 3, 2015
0349b5b
[SPARK-10411] [SQL] Move visualization above explain output and hide …
zsxwing Sep 3, 2015
67580f1
[SPARK-10332] [CORE] Fix yarn spark executor validation
holdenk Sep 3, 2015
3abc0d5
[SPARK-9596] [SQL] treat hadoop classes as shared one in IsolatedClie…
WangTaoTheTonic Sep 3, 2015
af0e312
[SPARK-8951] [SPARKR] support Unicode characters in collect()
Sep 3, 2015
49aff7b
[SPARK-10432] spark.port.maxRetries documentation is unclear
Sep 3, 2015
d911c68
[SPARK-10431] [CORE] Fix intermittent test failure. Wait for event qu…
Sep 3, 2015
754f853
[SPARK-9869] [STREAMING] Wait for all event notifications before asse…
Sep 3, 2015
e62f4a4
[SPARK-9672] [MESOS] Don’t include SPARK_ENV_LOADED when passing env …
pashields Sep 3, 2015
11ef32c
[SPARK-10430] [CORE] Added hashCode methods in AccumulableInfo and RD…
Sep 3, 2015
db4c130
[SPARK-9591] [CORE] Job may fail for exception during getting remote …
jeanlyn Sep 3, 2015
08b0750
[SPARK-10435] Spark submit should fail fast for Mesos cluster mode wi…
Sep 3, 2015
208fbca
[SPARK-10421] [BUILD] Exclude curator artifacts from tachyon dependen…
Sep 3, 2015
cf42138
[SPARK-10003] Improve readability of DAGScheduler
Sep 4, 2015
143e521
[MINOR] Minor style fix in SparkR
shivaram Sep 4, 2015
804a012
MAINTENANCE: Automated closing of pull requests.
marmbrus Sep 4, 2015
c3c0e43
[SPARK-10176] [SQL] Show partially analyzed plans when checkAnswer fa…
cloud-fan Sep 4, 2015
3339e6f
[SPARK-10450] [SQL] Minor improvements to readability / style / typos…
Sep 4, 2015
b087d23
[SPARK-9669] [MESOS] Support PySpark on Mesos cluster mode.
tnachen Sep 4, 2015
2e1c175
[SPARK-10454] [SPARK CORE] wait for empty event queue
Sep 4, 2015
eafe372
[SPARK-10311] [STREAMING] Reload appId and attemptId when app starts …
XuTingjun Sep 4, 2015
22eab70
[SPARK-10402] [DOCS] [ML] Add defaults to the scaladoc for params in ml/
holdenk Sep 5, 2015
47058ca
[SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correct…
yhuai Sep 5, 2015
6c75194
[HOTFIX] [SQL] Fixes compilation error
liancheng Sep 5, 2015
7a4f326
[SPARK-10440] [STREAMING] [DOCS] Update python API stuff in the progr…
tdas Sep 5, 2015
bca8c07
[SPARK-10434] [SQL] Fixes Parquet schema of arrays that may contain null
liancheng Sep 5, 2015
871764c
[SPARK-10013] [ML] [JAVA] [TEST] remove java assert from java unit tests
holdenk Sep 5, 2015
5ffe752
[SPARK-9767] Remove ConnectionManager.
rxin Sep 7, 2015
9d8e838
[DOC] Added R to the list of languages with "high-level API" support …
Sep 8, 2015
6ceed85
Docs small fixes
jaceklaskowski Sep 8, 2015
990c9f7
[SPARK-9170] [SQL] Use OrcStructInspector to be case preserving when …
viirya Sep 8, 2015
5b2192e
[SPARK-10480] [ML] Fix ML.LinearRegressionModel.copy()
yanboliang Sep 8, 2015
5fd5795
[SPARK-10316] [SQL] respect nondeterministic expressions in PhysicalO…
cloud-fan Sep 8, 2015
f7b55db
[SPARK-10470] [ML] ml.IsotonicRegressionModel.copy should set parent
yanboliang Sep 8, 2015
7a9dcbc
[SPARK-10441] [SQL] Save data correctly to json.
yhuai Sep 8, 2015
e6f8d36
[SPARK-10468] [ MLLIB ] Verify schema before Dataframe select API call
Sep 8, 2015
52b24a6
[SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentat…
tdas Sep 8, 2015
d637a66
[SPARK-10327] [SQL] Cache Table is not working while subquery has ali…
chenghao-intel Sep 8, 2015
2143d59
[HOTFIX] Fix build break caused by #8494
marmbrus Sep 8, 2015
ae74c3f
[RELEASE] Add more contributors & only show names in release notes.
rxin Sep 9, 2015
820913f
[SPARK-10071] [STREAMING] Output a warning when writing QueueInputDSt…
zsxwing Sep 9, 2015
52fe32f
[SPARK-9834] [MLLIB] implement weighted least squares via normal equa…
mengxr Sep 9, 2015
a157348
[SPARK-10464] [MLLIB] Add WeibullGenerator for RandomDataGenerator
yanboliang Sep 9, 2015
3a11e50
[SPARK-10373] [PYSPARK] move @since into pyspark from sql
Sep 9, 2015
0e2f216
[SPARK-10094] Pyspark ML Feature transformers marked as experimental
noel-smith Sep 9, 2015
2f6fd52
[SPARK-9654] [ML] [PYSPARK] Add IndexToString to PySpark
holdenk Sep 9, 2015
91a577d
[SPARK-10249] [ML] [DOC] Add Python Code Example to StopWordsRemover …
hhbyyh Sep 9, 2015
c1bc4f4
[SPARK-10227] fatal warnings with sbt on Scala 2.11
Sep 9, 2015
2ddeb63
[SPARK-10117] [MLLIB] Implement SQL data source API for reading LIBSV…
Lewuathe Sep 9, 2015
c0052d8
[SPARK-10481] [YARN] SPARK_PREPEND_CLASSES make spark-yarn related ja…
zjffdu Sep 9, 2015
71da163
[SPARK-10461] [SQL] make sure `input.primitive` is always variable na…
cloud-fan Sep 9, 2015
45de518
[SPARK-9730] [SQL] Add Full Outer Join support for SortMergeJoin
viirya Sep 9, 2015
56a0fe5
[SPARK-9772] [PYSPARK] [ML] Add Python API for ml.feature.VectorSlicer
yanboliang Sep 10, 2015
1dc7548
[MINOR] [MLLIB] [ML] [DOC] fixed typo: label for negative result shou…
sparadiso Sep 10, 2015
48817cc
[SPARK-10497] [BUILD] [TRIVIAL] Handle both locations for JIRAError w…
holdenk Sep 10, 2015
4f1daa1
[SPARK-10065] [SQL] avoid the extra copy when generate unsafe array
cloud-fan Sep 10, 2015
f892d92
[SPARK-7142] [SQL] Minor enhancement to BooleanSimplification Optimiz…
Sep 10, 2015
49da38e
[SPARK-10301] [SPARK-10428] [SQL] Addresses comments of PR #8583 and …
liancheng Sep 10, 2015
e048111
[SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill
chenghao-intel Sep 10, 2015
a76bde9
[SPARK-10469] [DOC] Try and document the three options
holdenk Sep 10, 2015
af3bc59
[SPARK-8167] Make tasks that fail from YARN preemption not fail job
mccheah Sep 10, 2015
f0562e8
[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecut…
dragos Sep 10, 2015
a5ef2d0
[SPARK-10514] [MESOS] waiting for min no of total cores acquired by S…
SleepyThread Sep 10, 2015
d88abb7
[SPARK-9990] [SQL] Create local hash join operator
zsxwing Sep 10, 2015
45e3be5
[SPARK-10049] [SPARKR] Support collecting data of ArraryType in DataF…
Sep 10, 2015
3db7255
[SPARK-10443] [SQL] Refactor SortMergeOuterJoin to reduce duplication
Sep 10, 2015
4204757
Add 1.5 to master branch EC2 scripts
shivaram Sep 10, 2015
89562a1
[SPARK-7544] [SQL] [PySpark] pyspark.sql.types.Row implements __getit…
yanboliang Sep 10, 2015
0eabea8
[SPARK-9043] Serialize key, value and combiner classes in ShuffleDepe…
massie Sep 11, 2015
339a527
[SPARK-10023] [ML] [PySpark] Unified DecisionTreeParams checkpointInt…
yanboliang Sep 11, 2015
a140dd7
[SPARK-10027] [ML] [PySpark] Add Python API missing methods for ml.fe…
yanboliang Sep 11, 2015
e1d7f64
[SPARK-10472] [SQL] Fixes DataType.typeName for UDT
liancheng Sep 11, 2015
9bbe33f
[SPARK-10556] Remove explicit Scala version for sbt project build files
ahirreddy Sep 11, 2015
c268ca4
[SPARK-10518] [DOCS] Update code examples in spark.ml user guide to u…
y-shimizu Sep 11, 2015
b656e61
[SPARK-10026] [ML] [PySpark] Implement some common Params for regress…
yanboliang Sep 11, 2015
b01b262
[SPARK-9773] [ML] [PySpark] Add Python API for MultilayerPerceptronCl…
yanboliang Sep 11, 2015
960d2d0
[SPARK-10537] [ML] document LIBSVM source options in public API doc a…
mengxr Sep 11, 2015
2e3a280
[MINOR] [MLLIB] [ML] [DOC] Minor doc fixes for StringIndexer and Meta…
jkbradley Sep 11, 2015
6ce0886
[SPARK-10540] [SQL] Ignore HadoopFsRelationTest's "test all data type…
yhuai Sep 11, 2015
5f46444
[SPARK-8530] [ML] add python API for MinMaxScaler
hhbyyh Sep 11, 2015
b231ab8
[SPARK-10546] Check partitionId's range in ExternalSorter#spill()
tedyu Sep 11, 2015
c373866
[PYTHON] Fixed typo in exception message
icaromedeiros Sep 11, 2015
d5d6473
[SPARK-10442] [SQL] fix string to boolean cast
cloud-fan Sep 11, 2015
1eede3b
[SPARK-7142] [SQL] Minor enhancement to BooleanSimplification Optimiz…
Sep 11, 2015
e626ac5
[SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK…
zsxwing Sep 11, 2015
c2af42b
[SPARK-9990] [SQL] Local hash join follow-ups
Sep 11, 2015
d74c6a1
[SPARK-10564] ThreadingSuite: assertion failures in threads don't fai…
Sep 11, 2015
c34fc19
[SPARK-9014] [SQL] Allow Python spark API to use built-in exponential…
0x0FFF Sep 11, 2015
6d83678
[SPARK-10566] [CORE] SnappyCompressionCodec init exception handling m…
dimfeld Sep 12, 2015
8285e3b
[SPARK-10554] [CORE] Fix NPE with ShutdownHook
Sep 12, 2015
22730ad
[SPARK-10547] [TEST] Streamline / improve style of Java API tests
srowen Sep 12, 2015
f4a2280
[SPARK-6548] Adding stddev to DataFrame functions
JihongMA Sep 12, 2015
b3a7480
[SPARK-10330] Add Scalastyle rule to require use of SparkHadoopUtil J…
JoshRosen Sep 12, 2015
1dc614b
[SPARK-10222] [GRAPHX] [DOCS] More thoroughly deprecate Bagel in favo…
srowen Sep 13, 2015
7d94924
Deprecates SQLConf.PARQUET_FOLLOW_PARQUET_FORMAT_SPEC
liancheng Sep 1, 2015
85bbfde
Removes instead of deprecates the old option
liancheng Sep 2, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[MINOR] [MLLIB] [ML] [DOC] Minor doc fixes for StringIndexer and Meta…
…dataUtils

Changes:
* Make Scala doc for StringIndexerInverse clearer.  Also remove Scala doc from transformSchema, so that the doc is inherited.
* MetadataUtils.scala: “ Helper utilities for tree-based algorithms” —> not just trees anymore

CC: holdenk mengxr

Author: Joseph K. Bradley <[email protected]>

Closes apache#8679 from jkbradley/doc-fixes-1.5.
  • Loading branch information
jkbradley authored and mengxr committed Sep 11, 2015
commit 2e3a280754a28dc36a71b9ff988e34cbf457f6c3
Original file line number Diff line number Diff line change
Expand Up @@ -181,10 +181,10 @@ class StringIndexerModel (

/**
* :: Experimental ::
* A [[Transformer]] that maps a column of string indices back to a new column of corresponding
* string values using either the ML attributes of the input column, or if provided using the labels
* supplied by the user.
* All original columns are kept during transformation.
* A [[Transformer]] that maps a column of indices back to a new column of corresponding
* string values.
* The index-string mapping is either from the ML attributes of the input column,
* or from user-supplied labels (which take precedence over ML attributes).
*
* @see [[StringIndexer]] for converting strings into indices
*/
Expand All @@ -202,32 +202,23 @@ class IndexToString private[ml] (
/** @group setParam */
def setOutputCol(value: String): this.type = set(outputCol, value)

/**
* Optional labels to be provided by the user, if not supplied column
* metadata is read for labels. The default value is an empty array,
* but the empty array is ignored and column metadata used instead.
* @group setParam
*/
/** @group setParam */
def setLabels(value: Array[String]): this.type = set(labels, value)

/**
* Param for array of labels.
* Optional labels to be provided by the user.
* Default: Empty array, in which case column metadata is used for labels.
* Optional param for array of labels specifying index-string mapping.
*
* Default: Empty array, in which case [[inputCol]] metadata is used for labels.
* @group param
*/
final val labels: StringArrayParam = new StringArrayParam(this, "labels",
"array of labels, if not provided metadata from inputCol is used instead.")
"Optional array of labels specifying index-string mapping." +
" If not provided or if empty, then metadata from inputCol is used instead.")
setDefault(labels, Array.empty[String])

/**
* Optional labels to be provided by the user, if not supplied column
* metadata is read for labels.
* @group getParam
*/
/** @group getParam */
final def getLabels: Array[String] = $(labels)

/** Transform the schema for the inverse transformation */
override def transformSchema(schema: StructType): StructType = {
val inputColName = $(inputCol)
val inputDataType = schema(inputColName).dataType
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.spark.sql.types.StructField


/**
* Helper utilities for tree-based algorithms
* Helper utilities for algorithms using ML metadata
*/
private[spark] object MetadataUtils {

Expand Down
16 changes: 8 additions & 8 deletions python/pyspark/ml/feature.py
Original file line number Diff line number Diff line change
Expand Up @@ -985,17 +985,17 @@ class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):
"""
.. note:: Experimental

A :py:class:`Transformer` that maps a column of string indices back to a new column of
corresponding string values using either the ML attributes of the input column, or if
provided using the labels supplied by the user.
All original columns are kept during transformation.
A :py:class:`Transformer` that maps a column of indices back to a new column of
corresponding string values.
The index-string mapping is either from the ML attributes of the input column,
or from user-supplied labels (which take precedence over ML attributes).
See L{StringIndexer} for converting strings into indices.
"""

# a placeholder to make the labels show up in generated doc
labels = Param(Params._dummy(), "labels",
"Optional array of labels to be provided by the user, if not supplied or " +
"empty, column metadata is read for labels")
"Optional array of labels specifying index-string mapping." +
" If not provided or if empty, then metadata from inputCol is used instead.")

@keyword_only
def __init__(self, inputCol=None, outputCol=None, labels=None):
Expand All @@ -1006,8 +1006,8 @@ def __init__(self, inputCol=None, outputCol=None, labels=None):
self._java_obj = self._new_java_obj("org.apache.spark.ml.feature.IndexToString",
self.uid)
self.labels = Param(self, "labels",
"Optional array of labels to be provided by the user, if not " +
"supplied or empty, column metadata is read for labels")
"Optional array of labels specifying index-string mapping. If not" +
" provided or if empty, then metadata from inputCol is used instead.")
kwargs = self.__init__._input_kwargs
self.setParams(**kwargs)

Expand Down