[SPARK-22397][ML]add multiple columns support to QuantileDiscretizer #19715

huaxingao · 2017-11-10T07:08:30Z

What changes were proposed in this pull request?

add multi columns support to QuantileDiscretizer.
When calculating the splits, we can either merge together all the probabilities into one array by calculating approxQuantiles on multiple columns at once, or compute approxQuantiles separately for each column. After doing the performance comparision, we found it’s better to calculating approxQuantiles on multiple columns at once.

Here is how we measuring the performance time:

    var duration = 0.0
    for (i<- 0 until 10) {
      val start = System.nanoTime()
      discretizer.fit(df)
      val end = System.nanoTime()
      duration += (end - start) / 1e9
    }
    println(duration/10)

Here is the performance test result:

numCols	NumRows	compute each approxQuantiles separately	compute multiple columns approxQuantiles at one time
10	60	0.3623195839	0.1626658607
10	6000	0.7537239841	0.3869370046
22	6000	1.6497598557	0.4767903059
50	6000	3.2268305752	0.7217818396

How was this patch tested?

add UT in QuantileDiscretizerSuite to test multi columns supports

huaxingao · 2017-11-10T07:10:41Z

@MLnick @viirya Could you please review? Thanks!

MLnick · 2017-11-10T12:39:25Z

Jenkins add to whitelist

SparkQA · 2017-11-10T12:52:52Z

Test build #83685 has finished for PR 19715 at commit 07bd868.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-11-11T19:31:19Z

Test build #83729 has finished for PR 19715 at commit 87ee0f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-11-12T06:35:30Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

 @Since("1.6.0")
 final class QuantileDiscretizer @Since("1.6.0") (@Since("1.6.0") override val uid: String)
-  extends Estimator[Bucketizer] with QuantileDiscretizerBase with DefaultParamsWritable {
+  extends Estimator[Bucketizer] with QuantileDiscretizerBase with DefaultParamsWritable


It looks a bit weird to have HasInputCols and HasOutputCols directly in QuantileDiscretizer and leave other params in QuantileDiscretizerBase.

But extending HasInputCols and HasOutputCols in QuantileDiscretizerBase causes binary compatibility issue. I think we don't want to break the compatibility in the effort of adding multi-col support.

I guess I will leave this as is even though it's a bit weird.

viirya · 2017-11-12T06:41:27Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+      }
+      bucketizer.setSplitsArray(distinctSplitsArray.toArray)
+      copyValues(bucketizer.setParent(this))
+    }


style issue:

... } else { ...

Will fix this. And fix the same problem in another place.

viirya · 2017-11-12T06:47:16Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val data = (0 until 100000).map { idx =>
+      (data1(idx), data2(idx))
+    }
+    val df: DataFrame = data.toSeq.toDF("input1", "input2")


nit: No need for the explicit type DataFrame.

Will remove DataFrame

viirya · 2017-11-12T06:49:26Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val data2 = Array.range(1, 200000, 2).map(_.toDouble)
+    val data = (0 until 100000).map { idx =>
+      (data1(idx), data2(idx))
+    }


val data seems just as data1.zip(data2)?

Yes. Will change to data1.zip(data2)

viirya · 2017-11-12T06:51:27Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val data2 = Array(1.0, 2.0, 3.0, 1.0, 1.0, 1.0, 1.0, 3.0, 2.0, 3.0, 1.0, 2.0)
+    val data = (0 until data1.length).map { idx =>
+      (data1(idx), data2(idx))
+    }


Use data1.zip(data2)?

Will change to data1.zip(data2).

viirya · 2017-11-12T06:51:50Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val data = (0 until data1.length).map { idx =>
+      (data1(idx), data2(idx))
+    }
+    val df: DataFrame = data.toSeq.toDF("input1", "input2")


nit: Remove DataFrame.

Will remove DataFrame.

viirya · 2017-11-12T09:16:16Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val numBucketsArray: Array[Int] = Array(2, 5, 10)
+    val data1 = Array.range(1, 21, 1).map(_.toDouble)
+    val expected1 = Array (0.0, 1.0, 1.0, 2.0, 2.0, 2.0, 3.0, 4.0, 4.0, 5.0,
+      5.0, 5.0, 6.0, 6.0, 7.0, 8.0, 8.0, 9.0, 9.0, 9.0)


Is this correct? I tried to apply the same data on current QuantileDiscretizer:

val data1 = Array.range(1, 21, 1).map(_.toDouble) val df = data1.toSeq.toDF val discretizer = new QuantileDiscretizer().setInputCol("value").setOutputCol("result").setNumBuckets(2) discretizer.fit(df).transform(df).show

+-----+------+ |value|result| +-----+------+ | 1.0| 0.0| | 2.0| 0.0| | 3.0| 0.0| | 4.0| 0.0| | 5.0| 0.0| | 6.0| 0.0| | 7.0| 0.0| | 8.0| 0.0| | 9.0| 0.0| | 10.0| 1.0| | 11.0| 1.0| | 12.0| 1.0| | 13.0| 1.0| | 14.0| 1.0| | 15.0| 1.0| | 16.0| 1.0| | 17.0| 1.0| | 18.0| 1.0| | 19.0| 1.0| | 20.0| 1.0| +-----+------+

I thought we are going to get all the probabilities derived from the numBucketsArray and use them for all the columns. In this case, all the probabilities for numBucketsArray (2,5,10) are (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). I am using these probabilities for all the input columns. In another word, I am using numsBuckets 10 for all the input columns. Is this right?

I think to set numBucketsArray, the first number of bucket is for first column, although we retrieve the approx-quantile for all probabilities at once.

AFractalThought · 2017-11-13T21:00:40Z

It would be great if multiple columns support could be extended to the some of the other transformers as well like StringIndexer.

viirya · 2017-11-14T08:10:38Z

@AFractalThought For StringIndexer, there is already SPARK-11215.

SparkQA · 2017-11-19T06:34:26Z

Test build #83994 has finished for PR 19715 at commit 5038e21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick

Sorry for the delay. Made a pass and some comments.

MLnick · 2017-11-29T12:26:13Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+  private[feature] def isQuantileDiscretizeMultipleColumns(): Boolean = {
+    if (isSet(inputCols) && isSet(inputCol)) {
+      logWarning("Both `inputCol` and `inputCols` are set, we ignore `inputCols` and this " +
+        "`QuantileDiscretize` only map one column specified by `inputCol`")


'only map' -> 'will only map'

MLnick · 2017-11-29T12:26:38Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+  private[feature] def isQuantileDiscretizeMultipleColumns(): Boolean = {
+    if (isSet(inputCols) && isSet(inputCol)) {
+      logWarning("Both `inputCol` and `inputCols` are set, we ignore `inputCols` and this " +
+        "`QuantileDiscretize` only map one column specified by `inputCol`")


QuantileDiscretize -> QuantileDiscretizer

MLnick · 2017-11-29T12:33:00Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+  val numBucketsArray = new IntArrayParam(this, "numBucketsArray", "Array of number of buckets " +
+    "(quantiles, or categories) into which data points are grouped. This is for multiple " +
+    "columns input. If numBucketsArray is not set but numBuckets is set, it means user wants " +
+    "to use the same numBuckets across all columns.")


Need a validator function here to ensure all bucket values >= 2

MLnick · 2017-11-29T12:33:53Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+   */
+  val numBucketsArray = new IntArrayParam(this, "numBucketsArray", "Array of number of buckets " +
+    "(quantiles, or categories) into which data points are grouped. This is for multiple " +
+    "columns input. If numBucketsArray is not set but numBuckets is set, it means user wants " +


"If transforming multiple columns and numBucketsArray is not set, but numBuckets is set, then numBuckets will be applied across all columns."

MLnick · 2017-11-29T12:37:19Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

 * categorical features. The number of bins can be set using the `numBuckets` parameter. It is
 * possible that the number of buckets used will be smaller than this value, for example, if there
 * are too few distinct values of the input to create enough distinct quantiles.
+ * Since 2.3.0,


Let's match the Bucketizer comment. So something like:

... Since 2.3.0, `QuantileDiscretizer ` can map multiple columns at once by setting the `inputCols` parameter. Note that when both the `inputCol` and `inputCols` parameters are set, a log warning will be printed and only `inputCol` will take effect, while `inputCols` will be ignored. To specify the number of buckets for each column , the `numBucketsArray ` parameter can be set, or if the number of buckets should be the same across columns, `numBuckets` can be set as a convenience.

MLnick · 2017-11-29T12:39:20Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

  def getNumBuckets: Int = getOrDefault(numBuckets)

+  /**
+   * Array of number of buckets (quantiles, or categories) into which data points are grouped.


Can add a comment about "each value must be greater than or equal to 2"

MLnick · 2017-11-29T12:45:53Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

-      (0.0 to 1.0 by 1.0/$(numBuckets)).toArray, $(relativeError))
+    val bucketizer = new Bucketizer(uid).setHandleInvalid($(handleInvalid))
+    if (isQuantileDiscretizeMultipleColumns) {
+      var bucketArray = Array.empty[Int]


val bucketSeq = if (isSet(numBucketsArray)) { $(numBucketsArray).toSeq } else { Seq($(numBuckets)) }

MLnick · 2017-11-29T13:27:15Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

-    val splits = dataset.stat.approxQuantile($(inputCol),
-      (0.0 to 1.0 by 1.0/$(numBuckets)).toArray, $(relativeError))
+    val bucketizer = new Bucketizer(uid).setHandleInvalid($(handleInvalid))
+    if (isQuantileDiscretizeMultipleColumns) {


This section overall seems like it can be cleaned up - it should be possible to have one code path for a Seq of numBuckets and at the end if transforming only one column the splits array should be the first element.

You could check the case of a single numBuckets value and Array.fill that value (if numBucketsArray is not set).

MLnick · 2017-11-29T13:30:25Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

    val model = discretizer.fit(df)
    assert(model.hasParent)
  }
+


We should add 2 tests:

test setting numBuckets is the same as setting numBucketsArray explicitly with identical values

test that QD over multiple columns produces the same results as 2x QDs over the same columns (as we did for Bucketizer)

MLnick · 2017-11-29T13:38:35Z

@huaxingao for posterity and recording purposes, could you post the performance comparison between the approach used here (of merging together all the probabilities into one array for approxQuantile vs computing separately) - you can post it on the JIRA.

SparkQA · 2017-11-30T23:36:57Z

Test build #84357 has finished for PR 19715 at commit 97ad483.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick

Made a pass and comment on fit. I will keep reviewing the test cases and revert with any further comments.

MLnick · 2017-12-04T07:17:14Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+ * `QuantileDiscretizer ` can map multiple columns at once by setting the `inputCols` parameter.
+ * Note that when both the `inputCol` and `inputCols` parameters are set, a log warning will be
+ * printed and only `inputCol` will take effect, while `inputCols` will be ignored. To specify
+ * the number of bucketsfor each column , the `numBucketsArray ` parameter can be set, or if the


"bucketsfor" -> "buckets for"

and remove the leading space from " number of buckets ..." on next line

MLnick · 2017-12-08T10:20:50Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+    val (inputColNames, outputColNames) = getInOutCols
+    val existingFields = schema.fields
+    var outputFields = existingFields
+    inputColNames.zip(outputColNames).map { case (inputColName, outputColName) =>


map can be foreach because there's no return value

MLnick · 2017-12-08T11:39:33Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

    transformSchema(dataset.schema, logging = true)
-    val splits = dataset.stat.approxQuantile($(inputCol),
-      (0.0 to 1.0 by 1.0/$(numBuckets)).toArray, $(relativeError))
+    val bucketizer = new Bucketizer(uid).setHandleInvalid($(handleInvalid))


Looking at this now, the Array.fill approach probably adds needless complexity.

But the multi-buckets case can perhaps still be cleaned up. How about something like this:

override def fit(dataset: Dataset[_]): Bucketizer = { transformSchema(dataset.schema, logging = true) val bucketizer = new Bucketizer(uid).setHandleInvalid($(handleInvalid)) if (isQuantileDiscretizeMultipleColumns) { val splitsArray = if (isSet(numBucketsArray)) { val probArrayPerCol = $(numBucketsArray).map { numOfBuckets => (0.0 to 1.0 by 1.0 / numOfBuckets).toArray } val probabilityArray = probArrayPerCol.flatten.sorted.distinct val splitsArrayRaw = dataset.stat.approxQuantile($(inputCols), probabilityArray, $(relativeError)) splitsArrayRaw.zip(probArrayPerCol).map { case (splits, probs) => val probSet = probs.toSet val idxSet = probabilityArray.zipWithIndex.collect { case (p, idx) if probSet(p) => idx }.toSet splits.zipWithIndex.collect { case (s, idx) if idxSet(idx) => s } } } else { dataset.stat.approxQuantile($(inputCols), (0.0 to 1.0 by 1.0 / $(numBuckets)).toArray, $(relativeError)) } bucketizer.setSplitsArray(splitsArray.map(getDistinctSplits)) } else { val splits = dataset.stat.approxQuantile($(inputCol), (0.0 to 1.0 by 1.0 / $(numBuckets)).toArray, $(relativeError)) bucketizer.setSplits(getDistinctSplits(splits)) } copyValues(bucketizer.setParent(this)) }

Then we don't need getSplitsForEachColumn method (or part of the above could be factored out into a private method if it makes sense).

huaxingao · 2017-12-09T01:41:46Z

@MLnick Thank you very much for your comments! I will change these.

SparkQA · 2017-12-09T03:05:31Z

Test build #84674 has finished for PR 19715 at commit 445bd84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

WeichenXu123 · 2017-12-11T08:50:53Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+
+  private[feature] def isQuantileDiscretizeMultipleColumns(): Boolean = {
+    if (isSet(inputCols) && isSet(inputCol)) {
+      logWarning("Both `inputCol` and `inputCols` are set, we ignore `inputCols` and this " +


According to the discussion result at JIRA SPARK-8418, we should throw exception when both inputCol and inputCols are specified ?

@WeichenXu123 I will change to throw Exception. Thanks.

SparkQA · 2017-12-12T06:30:28Z

Test build #84753 has finished for PR 19715 at commit 0e5971b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2017-12-12T10:20:10Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

-    if (!isQuantileDiscretizeMultipleColumns) {
+    require((isSet(inputCol) && isSet(outputCol) && !isSet(inputCols) && !isSet(outputCols)) ||
+      (!isSet(inputCol) && !isSet(outputCol) && isSet(inputCols) && isSet(outputCols)),
+      "Only allow to set either inputCol/outputCol, or inputCols/outputCols"


I think a better message is something like "QuantileDiscretizer only supports setting either ..."

MLnick · 2017-12-12T12:52:12Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

- *  number of buckets should be the same across columns, `numBuckets` can be set as a convenience.
+ * `QuantileDiscretizer` can map multiple columns at once by setting the `inputCols` parameter.
+ * Note that only one of `inputCol` and `inputCols` parameters can be set. If both of the
+ * `inputCol` and `inputCols` parameters are set, an Exception will be thrown. To specify the


Think we can simplify to "If both inputCol and inputCols are set, ..." (since we already said in the previous sentence that only one of the parameters can be set)

@MLnick Thank you very much for your comments. I will change these.

SparkQA · 2017-12-12T18:51:21Z

Test build #84780 has finished for PR 19715 at commit a030da1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick

Ok, made another pass. Overall tests look good. Made a few small comments.

As I think @jkbradley mentioned elsewhere, we should just check that we don't break save/load back compat (it should be ok but let's confirm).

So, if we create and save a QD before this PR and load it after this PR, it will still work fine.

MLnick · 2017-12-15T09:32:47Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+        val dataFrame: DataFrame = validData1.zip(validData2).zip(v).zip(w).map {
+          case (((a, b), c), d) => (a, b, c, d)
+        }.toSeq.toDF("input1", "input2", "expected1", "expected2")
+        dataFrame.show


remove the show call here

MLnick · 2017-12-15T09:33:02Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+        }.toSeq.toDF("input1", "input2", "expected1", "expected2")
+        dataFrame.show
+        val result = discretizer.fit(dataFrame).transform(dataFrame)
+        result.show


MLnick · 2017-12-15T09:36:39Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    import spark.implicits._
+
+    val datasetSize = 20
+    val numBucketsArray: Array[Int] = Array(2, 5, 10)


This is unused?

MLnick · 2017-12-15T09:36:56Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val data1 = Array.range(1, 21, 1).map(_.toDouble)
+    val data2 = Array.range(1, 40, 2).map(_.toDouble)
+    val data3 = Array.range(1, 60, 3).map(_.toDouble)
+    val data = (0 until 20).map { idx =>


can use datasetSize here? Or remove datasetSize as it's unused.

MLnick · 2017-12-15T09:43:08Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

+    val df = sc.parallelize(Array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0))
+      .map(Tuple1.apply).toDF("input")
+    // When both inputCol and inputCols are set, we throw Exception.
+    intercept[Exception] {


Maybe intercept IllegalArgumentException to be more specific.

@MLnick Thanks a lot for your comments. I will change these.

MLnick · 2017-12-15T09:44:27Z

mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala

+    if (isSet(inputCol)) {
+      (Array($(inputCol)), Array($(outputCol)))
+    } else {
+      require($(inputCols).length == $(outputCols).length,


We should add a small test case for mismatched sizes of inputCols / outputCols.

huaxingao · 2017-12-15T19:43:29Z

I have also verified the save/load back compatibility.
Thanks a lot for your comments! @MLnick

SparkQA · 2017-12-15T20:44:28Z

Test build #84974 has finished for PR 19715 at commit 99726a1.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2017-12-20T17:26:40Z

retest this please

SparkQA · 2017-12-20T18:28:43Z

Test build #85199 has finished for PR 19715 at commit 99726a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2017-12-21T10:11:39Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

-      discretizer.fit(df)
+  test("multiple columns: Both inputCol and inputCols are set") {
+    intercept[IllegalArgumentException] {
+      new QuantileDiscretizer().setInputCol("in").setInputCols(Array("in1", "in2")).getInOutCols


I think I slightly prefer to actually test that the error is thrown during transform

Thanks for spending so much time to review this PR. I will change this.

SparkQA · 2017-12-21T19:42:47Z

Test build #85275 has finished for PR 19715 at commit 486b68d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2017-12-29T20:08:57Z

Thanks for the changes @huaxingao. This LGTM now - any further comments from others?

MLnick · 2017-12-29T20:09:26Z

Jenkins retest this please

SparkQA · 2017-12-29T21:17:52Z

Test build #85524 has finished for PR 19715 at commit 486b68d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2017-12-31T12:39:18Z

Merged to master. If there are any further small comments / clean ups we can do that during QA for 2.3

Thanks @huaxingao and all others for review!

huaxingao · 2017-12-31T20:03:14Z

Thank you all for your help!!

[SPARK-22397][ML]add multiple columns support to QuantileDiscretizer

07bd868

fix binary compatibility issue

87ee0f3

viirya reviewed Nov 12, 2017

View reviewed changes

address comments

5038e21

MLnick suggested changes Nov 29, 2017

View reviewed changes

address comments

97ad483

MLnick reviewed Dec 8, 2017

View reviewed changes

address comments

445bd84

WeichenXu123 reviewed Dec 11, 2017

View reviewed changes

throw Exception if both inputCol and inputCols are set

0e5971b

MLnick reviewed Dec 12, 2017

View reviewed changes

Address Comments

a030da1

MLnick suggested changes Dec 15, 2017

View reviewed changes

Address Comments for test case

99726a1

MLnick reviewed Dec 21, 2017

View reviewed changes

address comment

486b68d

asfgit closed this in 3d8837e Dec 31, 2017

jkbradley mentioned this pull request Jun 13, 2018

[SPARK-23265][ML]Update multi-column error handling logic in QuantileDiscretizer #20442

Closed

[SPARK-22397][ML]add multiple columns support to QuantileDiscretizer #19715

[SPARK-22397][ML]add multiple columns support to QuantileDiscretizer #19715

Uh oh!

Conversation

huaxingao commented Nov 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

huaxingao commented Nov 10, 2017

Uh oh!

MLnick commented Nov 10, 2017

Uh oh!

SparkQA commented Nov 10, 2017

Uh oh!

SparkQA commented Nov 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AFractalThought commented Nov 13, 2017

Uh oh!

viirya commented Nov 14, 2017

Uh oh!

SparkQA commented Nov 19, 2017

Uh oh!

MLnick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MLnick commented Nov 29, 2017

Uh oh!

SparkQA commented Nov 30, 2017

Uh oh!

MLnick left a comment

Choose a reason for hiding this comment

huaxingao commented Nov 10, 2017 •

edited

Loading

MLnick Dec 12, 2017 •

edited

Loading

MLnick left a comment •

edited

Loading