Skip to content

[SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres #26412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 32 commits into from
Closed

[SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres #26412

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
0b293db
[SPARK-29774][SQL] Date and Timestamp type +/- null should be null as…
yaooqinn Nov 6, 2019
f726297
Merge branch 'master' into SPARK-29774
yaooqinn Nov 27, 2019
e7225a3
regen golden file
yaooqinn Nov 27, 2019
b925517
null - dates
yaooqinn Nov 27, 2019
cd49411
Merge branch 'master' into SPARK-29774
yaooqinn Dec 2, 2019
57b13e9
support +/-
yaooqinn Dec 2, 2019
eab6a83
support ×/÷
yaooqinn Dec 2, 2019
e8b75ba
import
yaooqinn Dec 2, 2019
02b3738
childResolved required
yaooqinn Dec 2, 2019
e89d806
regen golden file
yaooqinn Dec 2, 2019
0694e07
update comments
yaooqinn Dec 2, 2019
0f5618b
fix tests
yaooqinn Dec 3, 2019
efab3ec
fix tests
yaooqinn Dec 3, 2019
1c27be1
refine case match pattern
yaooqinn Dec 3, 2019
5df6980
fix ut
yaooqinn Dec 3, 2019
9817d2d
hack assert Equal
yaooqinn Dec 3, 2019
9808b9c
regen g f
yaooqinn Dec 3, 2019
b190612
AnalysisTest
yaooqinn Dec 3, 2019
e544137
regen g f
yaooqinn Dec 3, 2019
83705fd
fix test
yaooqinn Dec 4, 2019
846802d
date add/sub only work for int/smallint/tinyint
yaooqinn Dec 4, 2019
4af7edb
regen g f
yaooqinn Dec 4, 2019
a67be30
refine
yaooqinn Dec 4, 2019
9a1affd
type coercion for subtract timestamp
yaooqinn Dec 4, 2019
ae70022
add and reorgnize tests in datetime.sql
yaooqinn Dec 4, 2019
571225b
DateExpressionsSuite
yaooqinn Dec 4, 2019
6052e5a
fix py
yaooqinn Dec 4, 2019
928fd86
fix py
yaooqinn Dec 4, 2019
254d2d2
Revert "fix py"
yaooqinn Dec 5, 2019
5dd632c
fix py
yaooqinn Dec 5, 2019
c84d46e
rm unresolved binary arithmetic
yaooqinn Dec 5, 2019
a44948e
import
yaooqinn Dec 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-29774][SQL] Date and Timestamp type +/- null should be null as… 
… Postgres
  • Loading branch information
@yaooqinn
yaooqinn committed Nov 6, 2019
commit 0b293db1e2674523c13ff5cd906e92a921531826
5 changes: 5 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -856,14 +856,19 @@ object TypeCoercion {
DivideInterval(l, r)

case Add(l @ DateType(), r @ IntegerType()) => DateAdd(l, r)
case Add(l @ DateType(), r @ NullType()) => DateAdd(l, Cast(r, IntegerType))
case Add(l @ IntegerType(), r @ DateType()) => DateAdd(r, l)
case Add(l @ NullType(), r @ DateType()) => DateAdd(r, Cast(l, IntegerType))
case Subtract(l @ DateType(), r @ IntegerType()) => DateSub(l, r)
case Subtract(l @ DateType(), r @ NullType()) => DateSub(l, Cast(r, IntegerType))
case Subtract(l @ DateType(), r @ DateType()) =>
Copy link
Member

@maropu maropu Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we merge the multiple rule above into one like this?

      case b @ BinaryOperator(l @ DateType(), r @ NullType()) =>
        b.withNewChildren(Seq(l, Cast(r, IntegerType)))

Copy link
Member Author

@yaooqinn yaooqinn Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so, we might leave a trivial bug here if we set spark.sql.optimizer.maxIterations=1, it will not be transformed to DateAdd

Copy link
Member

@maropu maropu Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm..., I personally think that behaivour looks a little weired to me. Probably, the root cause is that Subtract(l @ DateType(), r @ NullType()).checkInputDataTypes.isSuccess returns true. To fix this issue, we might need to modify that check code to return false. cc: @cloud-fan

Copy link
Member Author

@yaooqinn yaooqinn Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, add with numeric type and null type is also handled in TypeCoercion too

if (SQLConf.get.usePostgreSQLDialect) DateDiff(l, r) else SubtractDates(l, r)
case Subtract(l @ TimestampType(), r @ TimestampType()) =>
SubtractTimestamps(l, r)
case Subtract(l @ TimestampType(), r @ DateType()) =>
SubtractTimestamps(l, Cast(r, TimestampType))
case Subtract(l @ TimestampType(), r @ NullType()) =>
Copy link
Contributor

@cloud-fan cloud-fan Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about null - timestamp?

Copy link
Member Author

@yaooqinn yaooqinn Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we need them too. checked with pg

SubtractTimestamps(l, Cast(r, NullType))
case Subtract(l @ DateType(), r @ TimestampType()) =>
SubtractTimestamps(Cast(l, TimestampType), r)
}
Expand Down
4 changes: 4 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/datetime.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ select date '2001-10-01' - 7;
select date '2001-10-01' - date '2001-09-28';
select date'2020-01-01' - timestamp'2019-10-06 10:11:12.345678';
select timestamp'2019-10-06 10:11:12.345678' - date'2020-01-01';
select date '2001-09-28' + null;
select date '2001-09-28' - null;
select null + date '2001-09-28';
select timestamp'2019-10-06 10:11:12.345678' - null;

-- interval operations
select 3 * (timestamp'2019-10-15 10:11:12.001002' - date'2019-10-15');
Expand Down
43 changes: 42 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/datetime.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 20
-- Number of queries: 25


-- !query 0
Expand Down Expand Up @@ -169,3 +169,44 @@ select (timestamp'2019-10-15' - timestamp'2019-10-14') / 1.5
struct<divide_interval(subtracttimestamps(TIMESTAMP('2019-10-15 00:00:00'), TIMESTAMP('2019-10-14 00:00:00')), CAST(1.5 AS DOUBLE)):interval>
-- !query 19 output
interval 16 hours


-- !query 20
select date '2001-09-28' + null
-- !query 20 schema
struct<date_add(DATE '2001-09-28', CAST(NULL AS INT)):date>
-- !query 20 output
NULL


-- !query 21
select date '2001-09-28' - null
-- !query 21 schema
struct<date_sub(DATE '2001-09-28', CAST(NULL AS INT)):date>
-- !query 21 output
NULL


-- !query 22
select null + date '2001-09-28'
-- !query 22 schema
struct<date_add(DATE '2001-09-28', CAST(NULL AS INT)):date>
-- !query 22 output
NULL


-- !query 23
select timestamp'2019-10-06 10:11:12.345678' - null
-- !query 23 schema
struct<subtracttimestamps(TIMESTAMP('2019-10-06 10:11:12.345678'), CAST(CAST(NULL AS NULL) AS TIMESTAMP)):interval>
-- !query 23 output
NULL


-- !query 24
select cast(k as date) + v, cast(k as date) - v from values ("2001-09-28", null), ("2001-09-27", 1) t(k, v)
-- !query 24 schema
struct<date_add(CAST(k AS DATE), v):date,date_sub(CAST(k AS DATE), v):date>
-- !query 24 output
2001-09-28 2001-09-26
NULL NULL