[SPARK-28637][SQL] Thriftserver support interval type #25277

wangyum · 2019-07-28T06:37:13Z

What changes were proposed in this pull request?

bin/spark-shell support query interval value:

scala> spark.sql("SELECT interval 3 months 1 hours AS i").show(false)
+-------------------------+
|i                        |
+-------------------------+
|interval 3 months 1 hours|
+-------------------------+

But sbin/start-thriftserver.sh can't support query interval value:

0: jdbc:hive2://localhost:10000/default> SELECT interval 3 months 1 hours AS i;
Error: java.lang.IllegalArgumentException: Unrecognized type name: interval (state=,code=0)

This PR maps CalendarIntervalType to StringType for TableSchema to make Thriftserver support query interval value because we do not support INTERVAL_YEAR_MONTH type and INTERVAL_DAY_TIME:

spark/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Type.java

Lines 73 to 78 in 02c3369

    
           INTERVAL_YEAR_MONTH_TYPE("INTERVAL_YEAR_MONTH", 
        
               java.sql.Types.OTHER, 
        
               TTypeId.INTERVAL_YEAR_MONTH_TYPE), 
        
           INTERVAL_DAY_TIME_TYPE("INTERVAL_DAY_TIME", 
        
               java.sql.Types.OTHER, 
        
               TTypeId.INTERVAL_DAY_TIME_TYPE),

SPARK-27791: Support SQL year-month INTERVAL type
SPARK-27793: Support SQL day-time INTERVAL type

How was this patch tested?

unit tests

wangyum · 2019-07-28T06:38:26Z

Will file a JIRA later.

wangyum · 2019-07-28T06:39:11Z

cc @juliuszsompolski

SparkQA · 2019-07-28T06:57:19Z

Test build #108266 has finished for PR 25277 at commit a2a35f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

...r/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala

juliuszsompolski · 2019-07-30T08:19:00Z

cc @bogdanghit

bogdanghit · 2019-07-30T15:37:31Z

Can you please add more details to the description of the PR?

juliuszsompolski · 2019-08-22T10:31:00Z

This has been filed as SPARK-28637. Could you update the PR description?
If we want to return intervals as string, since Spark does not support actual Interval Year Month and Interval Day Second, then I think this PR is good to go.

But then, I think maybe we should actually override GetTypeInfoOperation.
Currently !typeinfo returns INTERVAL_YEAR_MONTH, INTERVAL_DAY_TIME, ARRAY, MAP, STRUCT, UNIONTYPE and USER_DEFINED, all of which Spark turns into string.
Maybe we should make SparkGetTypeInfoOperation, to exclude types which we don't support?

Then we would be missing only SparkGetCatalogsOperation, which we may add for completeness - e.g. for it to use the HiveThriftServer2.listener and show up in the UI.

WDYT @wangyum ?

wangyum · 2019-08-22T12:41:14Z

Thank you @juliuszsompolski I will try to override GetTypeInfoOperation.

…ort-interval-type

SparkQA · 2019-08-25T15:32:36Z

Test build #109703 has finished for PR 25277 at commit d50f73c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-25T16:00:41Z

Test build #109704 has finished for PR 25277 at commit 28f226b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

juliuszsompolski · 2019-08-26T13:00:47Z

...r/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala

+      val attrTypeString = field.dataType match {
+        case NullType => "void"
+        case CalendarIntervalType => "string"
+        case other => other.catalogString


not related to Interval itself, but shouldn't it also return "string" for case ArrayType | _: StructType | _: MapType | _: UserDefinedType[_] here?

This is because Type does not have a type mapping to our CalendarIntervalType:

spark/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Type.java

Lines 31 to 105 in 02c3369

public enum Type {

NULL_TYPE("VOID",

java.sql.Types.NULL,

TTypeId.NULL_TYPE),

BOOLEAN_TYPE("BOOLEAN",

java.sql.Types.BOOLEAN,

TTypeId.BOOLEAN_TYPE),

TINYINT_TYPE("TINYINT",

java.sql.Types.TINYINT,

TTypeId.TINYINT_TYPE),

SMALLINT_TYPE("SMALLINT",

java.sql.Types.SMALLINT,

TTypeId.SMALLINT_TYPE),

INT_TYPE("INT",

java.sql.Types.INTEGER,

TTypeId.INT_TYPE),

BIGINT_TYPE("BIGINT",

java.sql.Types.BIGINT,

TTypeId.BIGINT_TYPE),

FLOAT_TYPE("FLOAT",

java.sql.Types.FLOAT,

TTypeId.FLOAT_TYPE),

DOUBLE_TYPE("DOUBLE",

java.sql.Types.DOUBLE,

TTypeId.DOUBLE_TYPE),

STRING_TYPE("STRING",

java.sql.Types.VARCHAR,

TTypeId.STRING_TYPE),

CHAR_TYPE("CHAR",

java.sql.Types.CHAR,

TTypeId.CHAR_TYPE,

true, false, false),

VARCHAR_TYPE("VARCHAR",

java.sql.Types.VARCHAR,

TTypeId.VARCHAR_TYPE,

true, false, false),

DATE_TYPE("DATE",

java.sql.Types.DATE,

TTypeId.DATE_TYPE),

TIMESTAMP_TYPE("TIMESTAMP",

java.sql.Types.TIMESTAMP,

TTypeId.TIMESTAMP_TYPE),

INTERVAL_YEAR_MONTH_TYPE("INTERVAL_YEAR_MONTH",

java.sql.Types.OTHER,

TTypeId.INTERVAL_YEAR_MONTH_TYPE),

INTERVAL_DAY_TIME_TYPE("INTERVAL_DAY_TIME",

java.sql.Types.OTHER,

TTypeId.INTERVAL_DAY_TIME_TYPE),

BINARY_TYPE("BINARY",

java.sql.Types.BINARY,

TTypeId.BINARY_TYPE),

DECIMAL_TYPE("DECIMAL",

java.sql.Types.DECIMAL,

TTypeId.DECIMAL_TYPE,

true, false, false),

ARRAY_TYPE("ARRAY",

java.sql.Types.ARRAY,

TTypeId.ARRAY_TYPE,

true, true),

MAP_TYPE("MAP",

java.sql.Types.JAVA_OBJECT,

TTypeId.MAP_TYPE,

true, true),

STRUCT_TYPE("STRUCT",

java.sql.Types.STRUCT,

TTypeId.STRUCT_TYPE,

true, false),

UNION_TYPE("UNIONTYPE",

java.sql.Types.OTHER,

TTypeId.UNION_TYPE,

true, false),

USER_DEFINED_TYPE("USER_DEFINED",

java.sql.Types.OTHER,

TTypeId.USER_DEFINED_TYPE,

true, false);

Otherwise:

Caused by: java.lang.IllegalArgumentException: Unrecognized type name: interval at org.apache.hive.service.cli.Type.getType(Type.java:169) at org.apache.hive.service.cli.TypeDescriptor.<init>(TypeDescriptor.java:53) at org.apache.hive.service.cli.ColumnDescriptor.<init>(ColumnDescriptor.java:53) at org.apache.hive.service.cli.TableSchema.<init>(TableSchema.java:52) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$.getTableSchema(SparkExecuteStatementOperation.scala:321) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.resultSchema$lzycompute(SparkExecuteStatementOperation.scala:70) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.resultSchema(SparkExecuteStatementOperation.scala:65) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.getResultSetSchema(SparkExecuteStatementOperation.scala:161) at org.apache.hive.service.cli.operation.OperationManager.getOperationResultSetSchema(OperationManager.java:209) at org.apache.hive.service.cli.session.HiveSessionImpl.getResultSetMetadata(HiveSessionImpl.java:773) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)

But the query actually returns a string for Arrays, Maps, Structs etc. Should it tell the client that it returns an ARRAY_TYPE or MAP_TYPE or STRUCT_TYPE if it in reality returns a string?

Ok. I understand now.
I was wondering how does it work that if you here create a TableSchema with a complex type like Array, and then this TableSchema is used through getResultSetSchema to create the resultRowSet, how does the RowSet know that we are in fact going to return String, and not an Array.
But it goes all the way down to Column in the constructed ColumnBasedSet in https://github.com/apache/spark/blob/master/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Column.java#L123 that assumes String for any non-primitive type. (or to conversions in ColumnValue for RowBasedSet in older protocol versions https://github.com/apache/spark/blob/master/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java#L198).

Now I understand. Thank you 👍

…ort-interval-type

juliuszsompolski

LGTM. Thank you!

wangyum · 2019-09-04T00:02:22Z

retest this please

SparkQA · 2019-09-04T00:30:29Z

Test build #110069 has finished for PR 25277 at commit 28f226b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-04T00:41:44Z

retest this please

SparkQA · 2019-09-04T01:10:45Z

Test build #110072 has finished for PR 25277 at commit 28f226b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-04T01:37:50Z

Test build #110074 has finished for PR 25277 at commit 59b3598.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-04T02:11:53Z

retest this please

SparkQA · 2019-09-04T02:37:22Z

Test build #110077 has finished for PR 25277 at commit 59b3598.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-09-12T17:46:29Z

Thanks! Merged to master.

## What changes were proposed in this pull request? `bin/spark-shell` support query interval value: ```scala scala> spark.sql("SELECT interval 3 months 1 hours AS i").show(false) +-------------------------+ |i | +-------------------------+ |interval 3 months 1 hours| +-------------------------+ ``` But `sbin/start-thriftserver.sh` can't support query interval value: ```sql 0: jdbc:hive2://localhost:10000/default> SELECT interval 3 months 1 hours AS i; Error: java.lang.IllegalArgumentException: Unrecognized type name: interval (state=,code=0) ``` This PR maps `CalendarIntervalType` to `StringType` for `TableSchema` to make Thriftserver support query interval value because we do not support `INTERVAL_YEAR_MONTH` type and `INTERVAL_DAY_TIME`: https://github.com/apache/spark/blob/02c33694c8254f69cb36c71c0876194dccdbc014/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Type.java#L73-L78 [SPARK-27791](https://issues.apache.org/jira/browse/SPARK-27791): Support SQL year-month INTERVAL type [SPARK-27793](https://issues.apache.org/jira/browse/SPARK-27793): Support SQL day-time INTERVAL type ## How was this patch tested? unit tests Closes apache#25277 from wangyum/Thriftserver-support-interval-type. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Xiao Li <[email protected]>

Thriftserver support interval type

a2a35f5

dongjoon-hyun added the SQL label Jul 28, 2019

HyukjinKwon reviewed Jul 29, 2019

View reviewed changes

...r/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala Outdated Show resolved Hide resolved

juliuszsompolski mentioned this pull request Aug 22, 2019

[SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation #25073

Closed

wangyum changed the title ~~[WIP] Thriftserver support interval type~~ [WIP][SPARK-28637][SQL] Thriftserver support interval type Aug 22, 2019

Invalid interval value

d50f73c

wangyum changed the title ~~[WIP][SPARK-28637][SQL] Thriftserver support interval type~~ [SPARK-28637][SQL] Thriftserver support interval type Aug 25, 2019

Merge remote-tracking branch 'upstream/master' into Thriftserver-supp…

6807933

…ort-interval-type

enable interval type

28f226b

juliuszsompolski reviewed Aug 26, 2019

View reviewed changes

Merge remote-tracking branch 'upstream/master' into Thriftserver-supp…

11a6ab4

…ort-interval-type

juliuszsompolski approved these changes Sep 3, 2019

View reviewed changes

wangyum changed the title ~~[SPARK-28637][SQL] Thriftserver support interval type~~ [SPARK-28637][SQL][test-hadoop3.2] Thriftserver support interval type Sep 4, 2019

"string" -> StringType.catalogString

59b3598

wangyum changed the title ~~[SPARK-28637][SQL][test-hadoop3.2] Thriftserver support interval type~~ [SPARK-28637][SQL] Thriftserver support interval type Sep 4, 2019

gatorsmile closed this in 4a3a6b6 Sep 9, 2019

wangyum deleted the Thriftserver-support-interval-type branch September 9, 2019 06:39

juliuszsompolski mentioned this pull request Sep 9, 2019

[SPARK-28982][SQL] Implementation Spark's own GetTypeInfoOperation #25694

Closed

juliuszsompolski mentioned this pull request Aug 27, 2020

[SPARK-32696][SQL][test-hive1.2][test-hadoop2.7]Get columns operation should handle interval column properly #29539

Closed

	INTERVAL_YEAR_MONTH_TYPE("INTERVAL_YEAR_MONTH",
	java.sql.Types.OTHER,
	TTypeId.INTERVAL_YEAR_MONTH_TYPE),
	INTERVAL_DAY_TIME_TYPE("INTERVAL_DAY_TIME",
	java.sql.Types.OTHER,
	TTypeId.INTERVAL_DAY_TIME_TYPE),

	public enum Type {
	NULL_TYPE("VOID",
	java.sql.Types.NULL,
	TTypeId.NULL_TYPE),
	BOOLEAN_TYPE("BOOLEAN",
	java.sql.Types.BOOLEAN,
	TTypeId.BOOLEAN_TYPE),
	TINYINT_TYPE("TINYINT",
	java.sql.Types.TINYINT,
	TTypeId.TINYINT_TYPE),
	SMALLINT_TYPE("SMALLINT",
	java.sql.Types.SMALLINT,
	TTypeId.SMALLINT_TYPE),
	INT_TYPE("INT",
	java.sql.Types.INTEGER,
	TTypeId.INT_TYPE),
	BIGINT_TYPE("BIGINT",
	java.sql.Types.BIGINT,
	TTypeId.BIGINT_TYPE),
	FLOAT_TYPE("FLOAT",
	java.sql.Types.FLOAT,
	TTypeId.FLOAT_TYPE),
	DOUBLE_TYPE("DOUBLE",
	java.sql.Types.DOUBLE,
	TTypeId.DOUBLE_TYPE),
	STRING_TYPE("STRING",
	java.sql.Types.VARCHAR,
	TTypeId.STRING_TYPE),
	CHAR_TYPE("CHAR",
	java.sql.Types.CHAR,
	TTypeId.CHAR_TYPE,
	true, false, false),
	VARCHAR_TYPE("VARCHAR",
	java.sql.Types.VARCHAR,
	TTypeId.VARCHAR_TYPE,
	true, false, false),
	DATE_TYPE("DATE",
	java.sql.Types.DATE,
	TTypeId.DATE_TYPE),
	TIMESTAMP_TYPE("TIMESTAMP",
	java.sql.Types.TIMESTAMP,
	TTypeId.TIMESTAMP_TYPE),
	INTERVAL_YEAR_MONTH_TYPE("INTERVAL_YEAR_MONTH",
	java.sql.Types.OTHER,
	TTypeId.INTERVAL_YEAR_MONTH_TYPE),
	INTERVAL_DAY_TIME_TYPE("INTERVAL_DAY_TIME",
	java.sql.Types.OTHER,
	TTypeId.INTERVAL_DAY_TIME_TYPE),
	BINARY_TYPE("BINARY",
	java.sql.Types.BINARY,
	TTypeId.BINARY_TYPE),
	DECIMAL_TYPE("DECIMAL",
	java.sql.Types.DECIMAL,
	TTypeId.DECIMAL_TYPE,
	true, false, false),
	ARRAY_TYPE("ARRAY",
	java.sql.Types.ARRAY,
	TTypeId.ARRAY_TYPE,
	true, true),
	MAP_TYPE("MAP",
	java.sql.Types.JAVA_OBJECT,
	TTypeId.MAP_TYPE,
	true, true),
	STRUCT_TYPE("STRUCT",
	java.sql.Types.STRUCT,
	TTypeId.STRUCT_TYPE,
	true, false),
	UNION_TYPE("UNIONTYPE",
	java.sql.Types.OTHER,
	TTypeId.UNION_TYPE,
	true, false),
	USER_DEFINED_TYPE("USER_DEFINED",
	java.sql.Types.OTHER,
	TTypeId.USER_DEFINED_TYPE,
	true, false);

[SPARK-28637][SQL] Thriftserver support interval type #25277

[SPARK-28637][SQL] Thriftserver support interval type #25277

Uh oh!

Conversation

wangyum commented Jul 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

wangyum commented Jul 28, 2019

Uh oh!

wangyum commented Jul 28, 2019

Uh oh!

SparkQA commented Jul 28, 2019

Uh oh!

Uh oh!

juliuszsompolski commented Jul 30, 2019

Uh oh!

bogdanghit commented Jul 30, 2019

Uh oh!

juliuszsompolski commented Aug 22, 2019

Uh oh!

wangyum commented Aug 22, 2019

Uh oh!

SparkQA commented Aug 25, 2019

Uh oh!

SparkQA commented Aug 25, 2019

Uh oh!

juliuszsompolski Aug 26, 2019

Choose a reason for hiding this comment

Uh oh!

wangyum Aug 27, 2019

Choose a reason for hiding this comment

Uh oh!

juliuszsompolski Aug 27, 2019

Choose a reason for hiding this comment

Uh oh!

wangyum Sep 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliuszsompolski Sep 3, 2019

Choose a reason for hiding this comment

Uh oh!

juliuszsompolski left a comment

Choose a reason for hiding this comment

Uh oh!

wangyum commented Sep 4, 2019

Uh oh!

SparkQA commented Sep 4, 2019

Uh oh!

wangyum commented Sep 4, 2019

Uh oh!

SparkQA commented Sep 4, 2019

Uh oh!

SparkQA commented Sep 4, 2019

Uh oh!

wangyum commented Sep 4, 2019

Uh oh!

SparkQA commented Sep 4, 2019

Uh oh!

gatorsmile commented Sep 12, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

wangyum commented Jul 28, 2019 •

edited

Loading

wangyum Sep 3, 2019 •

edited

Loading