Skip to content

[WIP][SPARK-29108][SQL] Add new module sql/thriftserver and add v11 thrift protocol #26221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 12 commits into from
Closed

[WIP][SPARK-29108][SQL] Add new module sql/thriftserver and add v11 thrift protocol #26221

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8b56400
Add new module and add v11 thrift protocol
AngersZhuuuu Oct 22, 2019
e30f686
add pom lincense
AngersZhuuuu Oct 23, 2019
4a50bc9
update code folder name
AngersZhuuuu Oct 23, 2019
7ffdc3b
all like hive
AngersZhuuuu Oct 23, 2019
b74d5e0
remove py/cpp/r/php
AngersZhuuuu Oct 23, 2019
fc2648f
Maven generate thrift source code
wangyum Oct 24, 2019
5365dcf
org.apache.thrift.tools:maven-thrift-plugin -> org.apache.thrift:thri…
wangyum Oct 25, 2019
95d8137
save some basic code
AngersZhuuuu Oct 26, 2019
4dc5c7e
Merge pull request #2 from wangyum/SPARK-29108
AngersZhuuuu Oct 27, 2019
24ce6d4
Merge branch 'SPARK-29018-V11' of https://github.com/AngersZhuuuu/spa…
AngersZhuuuu Oct 27, 2019
5efe8cb
Revert "save some basic code"
AngersZhuuuu Oct 27, 2019
6f7d48a
Update TCLIService.thrift
AngersZhuuuu Oct 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update TCLIService.thrift
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Oct 27, 2019
commit 6f7d48a987fd0cabef7e0cd4ee8af54197d66eaa
4 changes: 2 additions & 2 deletions sql/thriftserver/if/TCLIService.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@
// * Service names begin with the letter "T", use a capital letter for each
// new word (with no underscores), and end with the word "Service".

namespace java org.apache.spark.service.rpc.thrift
namespace cpp apache.spark.service.rpc.thrift
namespace java org.apache.spark.sql.thriftserver.cli.thrift
namespace cpp apache.spark.sql.thriftserver.cli.thrift

// List of protocol versions. A new token should be
// added to the end of this list every time a change is made.
Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@juliuszsompolski Ok with this package name?

Copy link
Contributor

@juliuszsompolski juliuszsompolski Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

New code in https://github.com/AngersZhuuuu/spark/tree/SPARK-29018-V11-STEP2

Current change list:

  1. Impelemnt Type in scala since spark con't support all type of hive
  2. Implement Service/AbstractService prepare for remove hive conf in future
  3. Construct RowSet with StructType and Row
  4. Implement HiveAuthFactory since between 1.2.1/2.3.5, their delegation token managerment changed. Impelment on DelegationTokenMnagerment by scala
  5. MV tableTypeString from SparkMetadataOperationUtils to SparkMetadataOperation
  6. Since there are tableTypeString in SparkMetadataOperation remove ClassicTypeMapping, HiveTableTypeMapping, TableTypeMapping and TableTypeMappingFactory
  7. Implement all operation for spark since it execute in different way
  8. Add new method GetQueryId, SetClientInfo for thrift version v11 in ThriftCLIService
  9. Add statementid to Operation for implement GetQueryId
  10. Remove GlobalHivercFileProcessor setFetchSize processGlobalInitFileetc

Still working on this.

Expand Down