Skip to content

[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with None triggers cryptic failure #8969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jasoncl wants to merge 5 commits into from
Closed

[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with None triggers cryptic failure #8969

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
362778c
Added support for JNull in Metadata. Error message now displays the a…
jasoncl Oct 2, 2015
6bb4fcc
fix scala style
jasoncl Oct 2, 2015
978fdc7
changed from putting a None to a null in case of JNull
jasoncl Oct 23, 2015
edbee08
code cleanup
jasoncl Oct 23, 2015
5524a92
remove changes from other JIRA work
jasoncl Oct 23, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Added support for JNull in Metadata. Error message now displays the a… 
…ctual unsupported type in the tuple instead of just tuple2
  • Loading branch information
@jasoncl
jasoncl committed Oct 2, 2015
commit 362778c2f9fd5a30959e6451017a1bdf428cb272
7 changes: 7 additions & 0 deletions python/pyspark/sql/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,13 @@ def test_struct_type(self):
except ValueError:
self.assertEqual(1, 1)

def test_metadata_null(self):
from pyspark.sql.types import StructType, StringType, StructField
schema = StructType([StructField("f1", StringType(), True, None),
StructField("f2", StringType(), True, {'a': None})])
rdd = self.sc.parallelize([['a', 'b'],['c', 'd']])
self.sqlCtx.createDataFrame(rdd,schema)

def test_save_and_load(self):
df = self.df
tmpPath = tempfile.mkdtemp()
Expand Down
8 changes: 7 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,9 @@ object Metadata {
throw new RuntimeException(s"Do not support array of type ${other.getClass}.")
}
}
case other =>
case (key, JNull) =>
builder.putNone(key)
case (key, other) =>
throw new RuntimeException(s"Do not support type ${other.getClass}.")
}
builder.build()
Expand All @@ -181,6 +183,8 @@ object Metadata {
JString(x)
case x: Metadata =>
toJsonValue(x.map)
case None =>
JNull
case other =>
throw new RuntimeException(s"Do not support type ${other.getClass}.")
}
Expand Down Expand Up @@ -228,6 +232,8 @@ class MetadataBuilder {
map ++= metadata.map
this
}
/** Puts a Long. */
Copy link
Contributor

@JoshRosen JoshRosen Oct 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is incorrect. Also, please leave an additional line of whitespace before this line.

def putNone(key: String): this.type = put(key, None)

/** Puts a Long. */
def putLong(key: String, value: Long): this.type = put(key, value)
Expand Down