Skip to content

[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with None triggers cryptic failure #8969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jasoncl wants to merge 5 commits into from
Closed

[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with None triggers cryptic failure #8969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
362778c
Added support for JNull in Metadata. Error message now displays the a…
jasoncl Oct 2, 2015
6bb4fcc
fix scala style
jasoncl Oct 2, 2015
978fdc7
changed from putting a None to a null in case of JNull
jasoncl Oct 23, 2015
edbee08
code cleanup
jasoncl Oct 23, 2015
5524a92
remove changes from other JIRA work
jasoncl Oct 23, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions python/pyspark/sql/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,13 @@ def test_struct_type(self):
except ValueError:
self.assertEqual(1, 1)

def test_metadata_null(self):
from pyspark.sql.types import StructType, StringType, StructField
schema = StructType([StructField("f1", StringType(), True, None),
StructField("f2", StringType(), True, {'a': None})])
rdd = self.sc.parallelize([["a", "b"], ["c", "d"]])
self.sqlCtx.createDataFrame(rdd, schema)

def test_save_and_load(self):
df = self.df
tmpPath = tempfile.mkdtemp()
Expand Down
7 changes: 6 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,9 @@ object Metadata {
throw new RuntimeException(s"Do not support array of type ${other.getClass}.")
}
}
case other =>
case (key, JNull) =>
builder.putNull(key)
case (key, other) =>
throw new RuntimeException(s"Do not support type ${other.getClass}.")
}
builder.build()
Expand Down Expand Up @@ -229,6 +231,9 @@ class MetadataBuilder {
this
}

/** Puts a null. */
def putNull(key: String): this.type = put(key, null)
Copy link
Contributor

@yhuai yhuai Jan 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like not worth adding a method since it is just used once?

Copy link
Contributor Author

@jasoncl jasoncl Jan 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is created for code organization and clarity purpose. This way it can be easily reused in the future.


/** Puts a Long. */
def putLong(key: String, value: Long): this.type = put(key, value)

Expand Down