Skip to content

[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LinhongLiu wants to merge 22 commits into from
Closed

[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
44d66d6
modify class CalendarInterval and CalendarIntervalSuite
LinhongLiu Oct 11, 2019
4979e1e
modify DateTimeUtils, DateTimeUtilsSuite, TemporalSequenceImpl
LinhongLiu Oct 12, 2019
f09b529
modify CalendarInterval related classes in sql/catalyst
LinhongLiu Oct 12, 2019
0db5b7a
fix test in sql/catalyst
LinhongLiu Oct 12, 2019
ce879c2
fix remaining tests
LinhongLiu Oct 14, 2019
4ee8354
fix failed tests
LinhongLiu Oct 15, 2019
3d954d6
code clean and more bug fix
LinhongLiu Oct 16, 2019
05042e8
address comments and fix tests
LinhongLiu Oct 16, 2019
4c31401
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 16, 2019
064be74
fix python
LinhongLiu Oct 18, 2019
211543f
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 18, 2019
0778f9a
use 24 hours = 1 day assumption in window, trigger and watermark
LinhongLiu Oct 20, 2019
3a6518a
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 20, 2019
4d8383c
streaming changes followup
LinhongLiu Oct 20, 2019
1ac157e
fix conflict
LinhongLiu Oct 20, 2019
aac92bd
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 26, 2019
076ce42
fix code sytle and test cases
LinhongLiu Oct 28, 2019
3d62a24
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 28, 2019
2edd8a0
change long,long,long to int,int,long when store CanlendarInterval in…
LinhongLiu Oct 28, 2019
4e97bc0
fix comment
LinhongLiu Oct 29, 2019
0e87e2d
Merge remote-tracking branch 'origin/master' into calendarinterval
LinhongLiu Oct 31, 2019
2f90189
address comments
LinhongLiu Nov 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix python
  • Loading branch information
@LinhongLiu
LinhongLiu committed Oct 18, 2019
commit 064be74359280d92b32cb8ec176900fafe0cee03
4 changes: 2 additions & 2 deletions python/pyspark/sql/tests/test_pandas_udf_grouped_map.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,7 +539,7 @@ def f(pdf):
pdf['result'] = [pdf['id']] * len(pdf)
return pdf

result = df.groupby('group', window('ts', '5 days')).apply(f)\
result = df.groupby('group', window('ts', '120 hours')).apply(f)\
.select('id', 'result').collect()
for r in result:
self.assertListEqual(expected[r[0]], r[1])
Expand Down Expand Up @@ -583,7 +583,7 @@ def f(key, pdf):
expected[id][1] == window_range))
return pdf.assign(result=is_expected)

result = df.groupby('group', window('ts', '5 days')).apply(f).select('result').collect()
result = df.groupby('group', window('ts', '120 hours')).apply(f).select('result').collect()

# Check that all group and window_range values from udf matched expected
self.assertTrue(all([r[0] for r in result]))
Expand Down