[SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources #20208

dongjoon-hyun · 2018-01-09T18:35:43Z

What changes were proposed in this pull request?

The reader schema is said to be evolved (or projected) when it changed after the data is written. The followings are already supported in file-based data sources. Note that partition columns are not maintained in files. In this PR, column means non-partition column.

Add a column
Hide a column
Change a column position
Change a column type (upcast)

This issue aims to guarantee users a backward-compatible read-schema test coverage on file-based data sources and to prevent future regressions by adding read schema tests explicitly.

Here, we consider safe changes without data loss. For example, data type change should be from small types to larger types like int-to-long, not vice versa.

As of today, in the master branch, file-based data sources have the following coverage.

File Format	Coverage	Note
TEXT	N/A	Schema consists of a single string column.
CSV	1, 2, 4
JSON	1, 2, 3, 4
ORC	1, 2, 3, 4	Native vectorized ORC reader has the widest coverage among ORC formats.
PARQUET	1, 2, 3

How was this patch tested?

Pass the Jenkins with newly added test suites.

dongjoon-hyun · 2018-01-09T18:37:12Z

Hi, @gatorsmile , @cloud-fan , @HyukjinKwon , @viirya .
Could you review this PR?

SparkQA · 2018-01-09T21:55:28Z

Test build #85865 has finished for PR 20208 at commit 499801e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-01-09T23:38:10Z

Also, ping @sameeragarwal , too.

dongjoon-hyun · 2018-01-10T15:38:28Z

Also, ping @rxin , too.

dongjoon-hyun · 2018-01-11T18:00:32Z

Retest this please.

SparkQA · 2018-01-11T21:14:08Z

Test build #85983 has finished for PR 20208 at commit 499801e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-01-13T16:27:33Z

Retest this please.

gatorsmile · 2018-01-13T17:45:30Z

We are working on the Spark 2.3 release. Could you ping us after the release?

dongjoon-hyun · 2018-01-13T18:01:40Z

Thank you for review, @gatorsmile .
Is there any concern about shaking 2.3 release? This is a test case only PR to build a clear consensus since Apache Spark 2.3.0. I think it's safe to be part of 2.3.0.

gatorsmile · 2018-01-13T18:14:16Z

Do not have enough review bandwidth on this test-only PRs before Spark 2.3 release

SparkQA · 2018-01-13T19:39:28Z

Test build #86099 has finished for PR 20208 at commit 499801e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-01-16T00:17:37Z

Retest this please.

SparkQA · 2018-01-16T03:32:04Z

Test build #86144 has finished for PR 20208 at commit 499801e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

SparkQA · 2018-01-17T12:34:07Z

Test build #86259 has finished for PR 20208 at commit 22eb772.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-01-17T16:17:58Z

Retest this please.

SparkQA · 2018-01-17T19:42:18Z

Test build #86281 has finished for PR 20208 at commit 22eb772.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait RemoveColumnEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

HyukjinKwon · 2018-01-20T12:19:34Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaEvolutionTest.scala

nit: `byte` -> byte or the opposite for consistency with the same instances.

HyukjinKwon · 2018-01-20T12:20:40Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaEvolutionTest.scala

Here seems many tests have some duplicated codes .. can we maybe do such as something like as below?

Seq(byteDF, ...).zip("byte").foreach { case (df, t) => test(s"boolean to $t") { spark.read .schema("col1 long") .format(format) .options(options) .load(path) checkAnswer(df4, longDF) } }

I am fine with any idea to deal with this duplication.

~~Ur, for this, when we put the variables (byteDF, ...) outside of test functions, it seems to cause SQLContext errors.~~ Never mind. I handled that as lazy variables.

HyukjinKwon · 2018-01-20T12:21:44Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaEvolutionTest.scala

I think we can do withTempPath.

HyukjinKwon · 2018-01-20T12:31:48Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaEvolutionTest.scala

Shall we leave the number given above in this comment like (case 1.).

HyukjinKwon · 2018-01-20T12:41:11Z

cc @sameeragarwal for reviewing too. I vaguely remember we had a talk about this before.

dongjoon-hyun · 2018-01-20T15:39:25Z

Thank you for review, @HyukjinKwon . I'll update like that.

SparkQA · 2018-01-20T19:26:00Z

Test build #86413 has finished for PR 20208 at commit e1d6f2a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-20T19:53:21Z

Test build #86414 has finished for PR 20208 at commit 29c281d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-21T20:38:16Z

Hi, @gatorsmile , @cloud-fan , @sameeragarwal , @HyukjinKwon .
The PR is ready for review again. Spark commit log seems to be a little quiet since yesterday.
Could you squeeze some time to give for this Schema Evolution suite? Thank you in advance for any advice!

HyukjinKwon · 2018-01-22T01:40:31Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaEvolutionTest.scala

@dongjoon-hyun, how do we guarantee schema change in Parquet and ORC?

I thought we (roughly) randomly pick up a file, read its footer and then use it. So, I was thinking we don't properly support this. It makes sense to Parquet with mergeSchema tho.

I think it's not even guaranteed in CSV too because we will rely on its header from one file.

Correct, and this is not about schema merging.
The final correct schema is given by users (or Hive).
In this PR, all schema is given by users, but for Hive tables, we uses the Hive Metastore Schema.

Ohaaa, the schema is explicitly set here. Sorry, I missed it.

gatorsmile · 2018-01-22T04:03:28Z

Will do it after 2.3 release

dongjoon-hyun · 2018-01-24T19:06:47Z

Retest this please.

SparkQA · 2018-01-24T21:03:02Z

Test build #86599 has finished for PR 20208 at commit 29c281d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-24T21:07:38Z

Retest this please.

SparkQA · 2018-01-25T00:34:13Z

Test build #86603 has finished for PR 20208 at commit 29c281d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-26T22:18:19Z

Retest this please.

dongjoon-hyun · 2018-01-26T22:24:07Z

Hi, @rxin , @cloud-fan , @sameeragarwal , @HyukjinKwon .

Could you give me some opinions about this PR? I know that Xiao Li is busy for this period, so I didn't ping him at this time. For me, this PR is important. Sorry for being annoying you guys.

dongjoon-hyun · 2018-03-25T09:36:25Z

I'll update like the followings.

Remove Remove a column part from the description parts (docs/testsuite file doc) while keeping the test cases.
Add a clear description about partition columns position rules.
Mention upcast for Change a column type part.

For docs/sql-programming-guide.md, I'll keep during review period.

SparkQA · 2018-05-22T00:59:35Z

Test build #90919 has finished for PR 20208 at commit e136bc3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait HideColumnAtTheEndEvolutionTest extends SchemaEvolutionTest
trait HideColumnInTheMiddleEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait ToStringTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

SparkQA · 2018-05-22T01:17:26Z

Test build #90920 has finished for PR 20208 at commit ea9047a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait HideColumnAtTheEndEvolutionTest extends SchemaEvolutionTest
trait HideColumnInTheMiddleEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait ToStringTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-05-23T00:21:47Z

Sorry for the delay. I updated the PR according to the comments, @gatorsmile .
Could you review this once more?

SparkQA · 2018-06-11T11:21:01Z

Test build #91654 has finished for PR 20208 at commit a8026b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait HideColumnAtTheEndEvolutionTest extends SchemaEvolutionTest
trait HideColumnInTheMiddleEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait ToStringTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

…ed data sources

dongjoon-hyun · 2018-07-09T04:14:47Z

Rebased to the master.

SparkQA · 2018-07-09T06:15:37Z

Test build #92731 has finished for PR 20208 at commit ebd239e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait HideColumnAtTheEndEvolutionTest extends SchemaEvolutionTest
trait HideColumnInTheMiddleEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait ToStringTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

dongjoon-hyun · 2018-07-09T08:09:00Z

Retest this please.

SparkQA · 2018-07-09T11:50:21Z

Test build #92744 has finished for PR 20208 at commit ebd239e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SchemaEvolutionTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnEvolutionTest extends SchemaEvolutionTest
trait HideColumnAtTheEndEvolutionTest extends SchemaEvolutionTest
trait HideColumnInTheMiddleEvolutionTest extends SchemaEvolutionTest
trait ChangePositionEvolutionTest extends SchemaEvolutionTest
trait BooleanTypeEvolutionTest extends SchemaEvolutionTest
trait ToStringTypeEvolutionTest extends SchemaEvolutionTest
trait IntegralTypeEvolutionTest extends SchemaEvolutionTest
trait ToDoubleTypeEvolutionTest extends SchemaEvolutionTest
trait ToDecimalTypeEvolutionTest extends SchemaEvolutionTest

gatorsmile · 2018-07-09T22:43:57Z

docs/sql-programming-guide.md

 when `path/to/table/gender=male` is the path of the data and
 users set `basePath` to `path/to/table/`, `gender` will be a partitioning column.

+### Schema Evolution


I still want to avoid using schema evolution in the doc or tests. Schema Projection might better. More importantly, you have to clarify that this only covers the read path.

What is the behavior in the write path when the physical and data schemas are different.

Thank you for review, @gatorsmile . I'll update like that.

For write operation, we cannot specify schema like read path, .schema. Spark writes the new file into the directory additionally or overwrites the directory.

dongjoon-hyun · 2018-07-10T21:22:18Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaSuite.scala

+/**
+ * All file-based data sources supports column addition and removal at the end.
+ */
+abstract class ReadSchemaSuite


@gatorsmile . Now, it becomes ReadSchemaSuite.

dongjoon-hyun · 2018-07-10T21:23:56Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaTest.scala

+import org.apache.spark.sql.test.{SharedSQLContext, SQLTestUtils}
+
+/**
+ * The reader schema is said to be evolved (or projected) when it changed after the data is


Here, I clearly mentioned read schema and used evolved and projected as general verbs.

dongjoon-hyun · 2018-07-10T21:25:41Z

@gatorsmile , @HyukjinKwon .
Could you review this again for Spark 2.4?

gatorsmile · 2018-07-10T21:49:32Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaTest.scala

+ *   | CSV          | 1, 2, 4      |                                                        |
+ *   | JSON         | 1, 2, 3, 4   |                                                        |
+ *   | ORC          | 1, 2, 3, 4   | Native vectorized ORC reader has the widest coverage.  |
+ *   | PARQUET      | 1, 2, 3      |                                                        |


Thanks for helping improve the test coverage! All the included test cases are positive. How about the negative test cases? What kind of errors you hit?

Yes. Right. Since the main purpose of this PR is preventing regressions, it consists of positive-only. The errors are case-by-case for each data sources.

For BooleanTypeTest example, Parquet raises higher exceptions due to ClassCastException (at the bottom). JSON raises Results do not match test case failures without exceptions.

Parquet

org.apache.spark.sql.execution.QueryExecutionException: Encounter error while reading parquet files. One possible cause: Parquet column cannot be converted in the corresponding files. Details: ... Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file file:/private/var/folders/dc/1pz9m69x14q_gw8t7m143t1c0000gn/T/spark-4b3d788b-1d7e-4ca2-9c01-88f639daf02f/part-00000-975391e5-1f1d-49f5-8e12-3213281618ed-c000.snappy.parquet ... Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableByte cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableBoolean

SparkQA · 2018-07-11T00:52:58Z

Test build #92828 has finished for PR 20208 at commit a7064ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T00:53:16Z

Test build #92827 has finished for PR 20208 at commit 767d7ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ReadSchemaTest extends QueryTest with SQLTestUtils with SharedSQLContext
trait AddColumnTest extends ReadSchemaTest
trait HideColumnAtTheEndTest extends ReadSchemaTest
trait HideColumnInTheMiddleTest extends ReadSchemaTest
trait ChangePositionTest extends ReadSchemaTest
trait BooleanTypeTest extends ReadSchemaTest
trait ToStringTypeTest extends ReadSchemaTest
trait IntegralTypeTest extends ReadSchemaTest
trait ToDoubleTypeTest extends ReadSchemaTest
trait ToDecimalTypeTest extends ReadSchemaTest

dongjoon-hyun · 2018-07-12T00:29:52Z

The test suite is designed like the following according to their features.

class CSVReadSchemaSuite
  extends ReadSchemaSuite
  with IntegralTypeTest
  with ToDoubleTypeTest
  with ToDecimalTypeTest
  with ToStringTypeTest {

  override val format: String = "csv"
}

To add a negative test case, we need to do something like with NoBooleanTypeTest. How do you think about that? @gatorsmile

dongjoon-hyun · 2018-07-12T16:43:19Z

Retest this please.

dongjoon-hyun · 2018-07-12T16:45:01Z

@gatorsmile . Please let me know if I need to do more.

SparkQA · 2018-07-12T20:19:02Z

Test build #92939 has finished for PR 20208 at commit a7064ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-07-12T21:08:23Z

@dongjoon-hyun This PR is to improve the test coverage. LGTM.

When the schema do not match with the schemas of underlying data source, the current error messages might be weird. This is a common issue, I think. Please submit a separate PR to improve the error handling in these cases?

gatorsmile · 2018-07-12T21:09:03Z

Thanks! Merged to master.

dongjoon-hyun · 2018-07-13T16:11:03Z

Thank you so much, @gatorsmile . Sure. I'll make a PR to improve error handling for that.

dongjoon-hyun · 2018-07-13T16:11:28Z

Also, thank you, @HyukjinKwon .

HyukjinKwon reviewed Jan 20, 2018

View reviewed changes

HyukjinKwon reviewed Jan 22, 2018

View reviewed changes

[SPARK-23007][SQL][TEST] Add schema evolution test suite for file-bas…

ebd239e

…ed data sources

gatorsmile reviewed Jul 9, 2018

View reviewed changes

Address comments

767d7ba

dongjoon-hyun changed the title ~~[SPARK-23007][SQL][TEST] Add schema evolution test suite for file-based data sources~~ [SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources Jul 10, 2018

fix comments

a7064ac

dongjoon-hyun commented Jul 10, 2018

View reviewed changes

gatorsmile reviewed Jul 10, 2018

View reviewed changes

asfgit closed this in 07704c9 Jul 12, 2018

dongjoon-hyun deleted the SPARK-SCHEMA-EVOLUTION branch July 13, 2018 16:11

[SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources #20208

[SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources #20208

Uh oh!

Conversation

dongjoon-hyun commented Jan 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jan 9, 2018

Uh oh!

SparkQA commented Jan 9, 2018

Uh oh!

dongjoon-hyun commented Jan 9, 2018

Uh oh!

dongjoon-hyun commented Jan 10, 2018

Uh oh!

dongjoon-hyun commented Jan 11, 2018

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

dongjoon-hyun commented Jan 13, 2018

Uh oh!

gatorsmile commented Jan 13, 2018

Uh oh!

dongjoon-hyun commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jan 13, 2018

Uh oh!

SparkQA commented Jan 13, 2018

Uh oh!

dongjoon-hyun commented Jan 16, 2018

Uh oh!

SparkQA commented Jan 16, 2018

Uh oh!

SparkQA commented Jan 17, 2018

Uh oh!

dongjoon-hyun commented Jan 17, 2018

Uh oh!

SparkQA commented Jan 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jan 20, 2018

Uh oh!

dongjoon-hyun commented Jan 20, 2018

Uh oh!

SparkQA commented Jan 20, 2018

Uh oh!

SparkQA commented Jan 20, 2018

Uh oh!

dongjoon-hyun commented Jan 21, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jan 22, 2018

Uh oh!

dongjoon-hyun commented Jan 24, 2018

Uh oh!

SparkQA commented Jan 24, 2018

Uh oh!

dongjoon-hyun commented Jan 24, 2018

Uh oh!

dongjoon-hyun commented Jan 9, 2018 •

edited

Loading

dongjoon-hyun commented Jan 13, 2018 •

edited

Loading

dongjoon-hyun Jan 20, 2018 •

edited

Loading

dongjoon-hyun Jan 22, 2018 •

edited

Loading

dongjoon-hyun commented Jan 26, 2018 •

edited

Loading

dongjoon-hyun Jul 10, 2018 •

edited

Loading

dongjoon-hyun commented Jul 12, 2018 •

edited

Loading