Skip to content

[SPARK-13244][SQL] Migrates DataFrame to Dataset #11443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 37 commits into from
Closed

[SPARK-13244][SQL] Migrates DataFrame to Dataset #11443

Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
ed932d2
Temporarily renames Dataset to DS
liancheng Mar 1, 2016
e59e940
Renames DataFrame to Dataset[T]
liancheng Mar 1, 2016
b357371
Fixes Java API compilation failures
liancheng Mar 1, 2016
3783e31
Fixes styling issues
liancheng Mar 1, 2016
a02a922
Fixes compilation failure introduced while rebasing
liancheng Mar 1, 2016
3db81f8
Temporarily disables MiMA check for convenience
liancheng Mar 1, 2016
f67f497
Fixes infinite recursion in Dataset constructor
liancheng Mar 1, 2016
f921583
Fixes test failures
liancheng Mar 3, 2016
fa22261
Migrates encoder stuff to the new Dataset
liancheng Mar 3, 2016
8cf5672
Makes some shape-keeping operations typed
liancheng Mar 5, 2016
712ee19
Adds collectRows() for Java API
liancheng Mar 6, 2016
c73b91f
Migrates joinWith operations
liancheng Mar 6, 2016
54cb36a
Migrates typed select
liancheng Mar 7, 2016
cbd7519
Renames typed groupBy to groupByKey
liancheng Mar 7, 2016
f1a2903
Migrates typed groupBy
liancheng Mar 7, 2016
15b4193
Migrates functional transformers
liancheng Mar 7, 2016
9aff0e2
Removes the old DS class and gets everything compiled
liancheng Mar 7, 2016
f053852
Fixes compilation error
liancheng Mar 7, 2016
3a7aff4
Row encoder should produce GenericRowWithSchema
liancheng Mar 8, 2016
9f8e0ad
Fixes compilation error after rebasing
liancheng Mar 8, 2016
bc081a9
Migrated Dataset.showString should handle typed objects
liancheng Mar 8, 2016
6b69f43
MapObjects should also handle DecimalType and DateType
liancheng Mar 8, 2016
29cb70f
Always use eager analysis
liancheng Mar 9, 2016
ba86e09
Fixes compilation error after rebasing
liancheng Mar 10, 2016
5727f48
Fixes compilation error after rebasing
liancheng Mar 10, 2016
cd63810
Temporarily ignores Python UDT test cases
liancheng Mar 10, 2016
4c8d139
fix python
cloud-fan Mar 10, 2016
cf0c112
Merge pull request #9 from cloud-fan/ds-to-df
liancheng Mar 10, 2016
91942cf
fix pymllib
cloud-fan Mar 10, 2016
736fbcb
Merge pull request #10 from cloud-fan/ds-to-df
liancheng Mar 10, 2016
488a82e
MIMA
yhuai Mar 10, 2016
343c611
Fix typo...
yhuai Mar 10, 2016
63d4d69
MIMA: Exclude DataFrame class.
yhuai Mar 10, 2016
f6bcd50
Revert "MIMA: Exclude DataFrame class."
yhuai Mar 10, 2016
49c6fc2
Revert "Fix typo..."
yhuai Mar 10, 2016
d52ce17
Revert "MIMA"
yhuai Mar 10, 2016
7d29c06
Merge remote-tracking branch 'upstream/master' into ds-to-df
yhuai Mar 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert "MIMA" 
This reverts commit 488a82e.
  • Loading branch information
@yhuai
yhuai committed Mar 10, 2016
commit d52ce17f3c34293be2c0f8dec17fccc814483626
9 changes: 5 additions & 4 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,10 +561,11 @@ def main():
# spark build
build_apache_spark(build_tool, hadoop_version)

# backwards compatibility checks
if build_tool == "sbt":
# Note: compatiblity tests only supported in sbt for now
detect_binary_inop_with_mima()
# TODO Temporarily disable MiMA check for DF-to-DS migration prototyping
# # backwards compatibility checks
# if build_tool == "sbt":
# # Note: compatiblity tests only supported in sbt for now
# detect_binary_inop_with_mima()

# run the test suites
run_scala_tests(build_tool, hadoop_version, test_modules, excluded_tags)
Expand Down