Skip to content

[SPARK-12300][SQL][PYSPARK] fix schema inferance on local collections #10275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 6 commits into from
Closed

[SPARK-12300][SQL][PYSPARK] fix schema inferance on local collections #10275

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
935c167
Look at all elements of the local collection when infering the types
holdenk Dec 12, 2015
7ca7eca
Add a test
holdenk Dec 12, 2015
e01cbb4
Merge branch 'master' into SPARK-12300-fix-schmea-inferance-on-local-…
holdenk Dec 12, 2015
6e463fd
Fix test
holdenk Dec 12, 2015
b863cc9
python 3 import reduce from functools
holdenk Dec 12, 2015
3f2b825
Merge branch 'master' into SPARK-12300-fix-schmea-inferance-on-local-…
holdenk Dec 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add a test
  • Loading branch information
@holdenk
holdenk committed Dec 12, 2015
commit 7ca7ecad1faea3b3746aec10130ebf07f4e2a3ad
10 changes: 10 additions & 0 deletions python/pyspark/sql/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,16 @@ def test_apply_schema_to_row(self):
df3 = self.sqlCtx.createDataFrame(rdd, df.schema)
self.assertEqual(10, df3.count())

def test_infer_schema_to_local(self):
input = [{"a": 1}, {"b": "coffee"}]
df = self.sqlCtx.createDataFrame(input)
df2 = self.sqlCtx.createDataFrame(sc.parallelize(input), samplingRatio=1.0)
self.assertEqual(df.schema(), df2.schema())

rdd = self.sc.parallelize(range(10)).map(lambda x: Row(a=x))
df3 = self.sqlCtx.createDataFrame(rdd, df.schema)
self.assertEqual(10, df3.count())

def test_serialize_nested_array_and_map(self):
d = [Row(l=[Row(a=1, b='s')], d={"key": Row(c=1.0, d="2")})]
rdd = self.sc.parallelize(d)
Expand Down