Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
890 commits
Select commit Hold shift + click to select a range
27d0c3f
[SPARK-29139][CORE][TESTS] Increase timeout to wait for executor(s) t…
HeartSaVioR Sep 20, 2019
42050c3
[SPARK-27659][PYTHON] Allow PySpark to prefetch during toLocalIterator
holdenk Sep 20, 2019
4080c4b
[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness
holdenk Sep 20, 2019
8c8016a
[SPARK-21045][PYTHON] Allow non-ascii string as an exception message …
advancedxy Sep 20, 2019
abc88de
[SPARK-29063][SQL] Modify fillValue approach to support joined dataframe
xuanyuanking Sep 20, 2019
4a89fa1
[SPARK-29196][DOCS] Add JDK11 support to the document
dongjoon-hyun Sep 20, 2019
c764dd6
[SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with ne…
zhengruifeng Sep 21, 2019
ff3a737
[SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results sepa…
dongjoon-hyun Sep 21, 2019
252b6cf
[SPARK-29187][SQL] Return null from `date_part()` for the null `field`
MaxGekk Sep 21, 2019
a9ae262
[SPARK-28772][BUILD][MLLIB] Update breeze to 1.0
srowen Sep 21, 2019
f7cc695
[SPARK-29140][SQL] Handle parameters having "array" of javaType prope…
HeartSaVioR Sep 21, 2019
076186e
[SPARK-19147][CORE] Gracefully handle error in task after executor is…
Sep 21, 2019
93ac4e1
[SPARK-29053][WEBUI] Sort does not work on some columns
amanomer Sep 21, 2019
81b6f11
[SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event lo…
HeartSaVioR Sep 21, 2019
3e26492
[SPARK-29199][INFRA] Add linters and license/dependency checkers to G…
dongjoon-hyun Sep 21, 2019
c7c6b64
[SPARK-29121][ML][MLLIB] Support for dot product operation on Vector(s)
phpisciuneri Sep 21, 2019
3be5741
[SPARK-29190][SQL] Optimize `extract`/`date_part` for the millisecond…
MaxGekk Sep 22, 2019
89bad26
[SPARK-29200][SQL] Optimize `extract`/`date_part` for epoch
MaxGekk Sep 22, 2019
e2c4787
[CORE][MINOR] Correct a log message in DAGScheduler
Sep 22, 2019
051e691
[SPARK-28141][SQL] Support special date values
MaxGekk Sep 22, 2019
a838dbd
[SPARK-27463][PYTHON][FOLLOW-UP] Run the tests of Cogrouped pandas UDF
HyukjinKwon Sep 22, 2019
fe4bee8
[SPARK-29162][SQL] Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AngersZhuuuu Sep 22, 2019
76bc9db
[SPARK-29191][TESTS][SQL] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun Sep 22, 2019
51d3509
[SPARK-28599][SQL] Fix `Execution Time` and `Duration` column sorting…
wangyum Sep 22, 2019
7a2ea58
[SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQuer…
maropu Sep 22, 2019
f725d47
[SPARK-25341][CORE] Support rolling back a shuffle map stage and re-g…
xuanyuanking Sep 23, 2019
655356e
[SPARK-28892][SQL] support UPDATE in the parser and add the correspon…
Sep 23, 2019
c08bc37
[SPARK-29177][CORE] fix zombie tasks after stage abort
adrian-wang Sep 23, 2019
d22768a
[SPARK-29036][SQL] SparkThriftServer cancel job after execute() threa…
AngersZhuuuu Sep 23, 2019
0c40b94
[SPARK-29203][SQL][TESTS] Reduce shuffle partitions in SQLQueryTestSuite
wangyum Sep 23, 2019
d50f6e6
[SPARK-25903][CORE] TimerTask should be synchronized on ContextBarrie…
viirya Sep 23, 2019
c38f459
[SPARK-29016][BUILD] Update LICENSE and NOTICE for Hive 2.3
wangyum Sep 23, 2019
4513f1c
[SPARK-26848][SQL][SS] Introduce new option to Kafka source: offset b…
HeartSaVioR Sep 24, 2019
da7e5c4
[SPARK-19917][SQL] qualified partition path stored in catalog
windpiger Sep 24, 2019
7c85968
[SPARK-29218][WEBUI] Increase `Show Additional Metrics` checkbox widt…
sarutak Sep 24, 2019
cb72b10
[SPARK-29168][WEBUI] Use a unique color on selected item on timeline …
TomokoKomiyama Sep 24, 2019
b8b67ae
[SPARK-28527][SQL][TEST] Enable ThriftServerQueryTestSuite
wangyum Sep 24, 2019
81de9d3
[SPARK-28678][DOC] Specify that array indices start at 1 for function…
sheepstop Sep 24, 2019
7c02c14
[SPARK-28292][SQL] Enable Injection of User-defined Hint
gatorsmile Sep 24, 2019
fff2e84
[SPARK-29095][ML] add extractInstances
zhengruifeng Sep 24, 2019
b3e9be4
[SPARK-29229][SQL] Change the additional remote repository in Isolate…
xuanyuanking Sep 24, 2019
d75588c
[SPARK-29082][CORE] Skip delegation token generation if no credential…
gaborgsomogyi Sep 24, 2019
e650f8f
[SPARK-29230][CORE][TEST] Fix NPE in ProcfsMetricsGetterSuite
sev7e0 Sep 24, 2019
eef3abb
[SPARK-29226][BUILD] Upgrade jackson-databind to 2.9.10 and fix vulne…
beliefer Sep 25, 2019
d8b0914
[SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties int…
WeichenXu123 Sep 25, 2019
a36a723
[SPARK-29215][SQL] current namespace should be tracked in SessionCata…
cloud-fan Sep 25, 2019
58989cd
[SPARK-29168][WEBUI][FOLLOW-UP] Use a dark colors on selected Executo…
TomokoKomiyama Sep 25, 2019
e07cbbe
[SPARK-29236][CORE] Access 'executorDataMap' out of 'DriverEndpoint' …
ConeyLiu Sep 25, 2019
a1b90bf
[SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycl…
HeartSaVioR Sep 25, 2019
21db2f8
[SPARK-29237][SQL] Prevent real function names in expression example …
MaxGekk Sep 25, 2019
6a4235a
[SPARK-29249][SQL] V2 writer: Don't allow tableProperty for existing …
rdblue Sep 26, 2019
b8b59d6
[SPARK-29239][SPARK-29221][SQL] Subquery should not cause NPE when el…
viirya Sep 26, 2019
c8159c7
[SPARK-29197][SQL] Remove saveModeForDSV2 from DataFrameWriter
brkyvz Sep 26, 2019
66c9dc3
[SPARK-29255][SQL][TESTS] Rename package pgSQL to postgreSQL
gengliangwang Sep 26, 2019
a1213d5
[SPARK-28997][SQL] Add `spark.sql.dialect`
gengliangwang Sep 26, 2019
8beb736
[SPARK-29256][DOCS] Fix typo in building document
TomokoKomiyama Sep 26, 2019
d3679a9
[SPARK-27748][SS][FOLLOWUP] Correct the order of logging token as deb…
HeartSaVioR Sep 26, 2019
103de96
[SPARK-29202][DEPLOY] Driver java options are not passed to driver pr…
sandeep-katta Sep 26, 2019
9f3c821
[SPARK-29259][SQL] call fs.exists only when necessary
rahij Sep 26, 2019
bdc4943
[SPARK-29142][PYTHON][ML] Pyspark clustering models support column se…
huaxingao Sep 27, 2019
cd04607
[SPARK-29246][CORE] Remove unnecessary imports in `core` module
sev7e0 Sep 27, 2019
570525f
[SPARK-27715][SQL][UI] SQL query details in UI does not show in corre…
uncleGen Sep 27, 2019
ada3ad3
[SPARK-29175][SQL] Make additional remote maven repository in Isolate…
xuanyuanking Sep 27, 2019
aed7ff3
[SPARK-29258][ML][PYSPARK] parity between ml.evaluator and mllib.metrics
zhengruifeng Sep 27, 2019
bd28e8e
[SPARK-29213][SQL] Generate extra IsNotNull predicate in FilterExec
wangshuo128 Sep 27, 2019
4dd0066
[SPARK-21914][SQL][TESTS] Check results of expression examples
MaxGekk Sep 27, 2019
cc852d4
[SPARK-29015][SQL][TEST-HADOOP3.2] Reset class loader after initializ…
AngersZhuuuu Sep 27, 2019
4bffcf5
[SPARK-29275][SQL][DOC] Describe special date/timestamp values in the…
MaxGekk Sep 27, 2019
fda0e6e
[SPARK-29240][PYTHON] Pass Py4J column instance to support PySpark co…
HyukjinKwon Sep 27, 2019
420abb4
[SPARK-29263][SCHEDULER] Update `availableSlots` in `resourceOffers()…
juliuszsompolski Sep 27, 2019
233c214
[SPARK-29070][CORE] Make SparkLauncher log full spark-submit command …
jeff303 Sep 27, 2019
d72f398
[SPARK-27254][SS] Cleanup complete but invalid output files in Manife…
HeartSaVioR Sep 27, 2019
ece4213
[SPARK-21914][FOLLOWUP][TEST-HADOOP3.2][TEST-JAVA11] Clone SparkSessi…
MaxGekk Sep 28, 2019
94946e4
[SPARK-29281][SQL] Correct example of Like/RLike to test the origin i…
HeartSaVioR Sep 28, 2019
2409320
[SPARK-29237][SQL][FOLLOWUP] Ignore `SET` commands in expression exam…
MaxGekk Sep 28, 2019
8167714
[SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive …
wangyum Sep 28, 2019
67d5b9b
[SPARK-29172][SQL] Fix some exception issue of explain commands
TomokoKomiyama Sep 29, 2019
3170011
[SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
wangyum Sep 29, 2019
1d4b2f0
[SPARK-29247][SQL] Redact sensitive information in when construct Hiv…
AngersZhuuuu Sep 29, 2019
df28671
[SPARK-29282][TESTS] Use the same VM configurations for test/benchmark
dongjoon-hyun Sep 29, 2019
3ea9d68
[SPARK-29019][WEBUI] Improve tooltip JDBC/ODBC Server tab
planga82 Sep 29, 2019
dd92e15
[SPARK-29186][SQL] AliasIdentifier should be converted to Json in pre…
viirya Sep 30, 2019
989b0c7
[SPARK-29297][TESTS] Compare `core`/`mllib` module benchmarks in JDK8/11
dongjoon-hyun Sep 30, 2019
28b8383
[SPARK-29289][BUILD] Update scalatest, scalacheck, scopt, clapper, sc…
srowen Sep 30, 2019
1018390
[SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities
beliefer Sep 30, 2019
39eb79a
[SPARK-28074][SS] Log warn message on possible correctness issue for …
HeartSaVioR Sep 30, 2019
76791b8
[SPARK-27463][PYTHON][FOLLOW-UP] Miscellaneous documentation and code…
d80tb7 Sep 30, 2019
e1ea806
[SPARK-29291][CORE][SQL][STREAMING][MLLIB] Change procedure-like decl…
srowen Sep 30, 2019
a0b3d7a
[SPARK-29300][TESTS] Compare `catalyst` and `avro` module benchmark i…
dongjoon-hyun Oct 1, 2019
d841b33
[SPARK-25153][SQL] Improve error messages for columns with dots/periods
jeff303 Oct 1, 2019
bd031c2
[SPARK-29307][BUILD][TESTS] Remove scalatest deprecation warnings
dongjoon-hyun Oct 1, 2019
0cf2f48
[SPARK-29022][SQL] Fix SparkSQLCLI can not add jars by AddJarCommand
AngersZhuuuu Oct 1, 2019
a4601cb
[SPARK-29055][CORE] Update driver/executors' storage memory when bloc…
HeartSaVioR Oct 1, 2019
0cd436b
[SPARK-29244][CORE] Prevent freed page in BytesToBytesMap free again
viirya Oct 1, 2019
e138801
[SPARK-29311][SQL] Return seconds with fraction from `date_part()` an…
MaxGekk Oct 2, 2019
2ec3265
[MINOR][BUILD] Decode output of commands during merge script as UTF-8…
srowen Oct 2, 2019
c6938ea
[SPARK-29310][CORE][TESTS] TestMemoryManager should implement getExec…
JoshRosen Oct 2, 2019
3b1674c
[SPARK-29313][SQL] Fix failure on writing to `noop` in benchmarks
MaxGekk Oct 2, 2019
f2ead4d
[SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2
imback82 Oct 2, 2019
85dafab
[SPARK-29273][CORE] Save peakExecutionMemory value when writing task …
012huang Oct 2, 2019
9a84fae
[SPARK-29332][BUILD] Update zstd-jni to 1.4.3-1
dongjoon-hyun Oct 2, 2019
730a178
[SPARK-27297][SQL] Add higher order functions to scala API
nvander1 Oct 2, 2019
51d6ba7
[SPARK-28962][SQL] Provide index argument to filter lambda functions
henrydavidge Oct 2, 2019
e44d191
[SPARK-29322][CORE] Enable closeFrameOnFlush on ZstdOutputStream for …
HeartSaVioR Oct 3, 2019
178a1f3
[SPARK-29305][BUILD] Update LICENSE and NOTICE for Hadoop 3.2
AngersZhuuuu Oct 3, 2019
40485f4
[SPARK-29317][SQL][PYTHON] Avoid inheritance hierarchy in pandas CoGr…
HyukjinKwon Oct 3, 2019
ee66890
[SPARK-28084][SQL] Resolving the partition column name based on the r…
sujith71955 Oct 3, 2019
858bf76
[SPARK-29142][PYTHON][ML][FOLLOWUP][DOC] Replace incorrect :py:attr: …
zero323 Oct 3, 2019
2bc3fff
[SPARK-29341][PYTHON] Upgrade cloudpickle to 1.0.0
viirya Oct 3, 2019
7aca0dd
[SPARK-29296][BUILD][CORE] Remove use of .par to make 2.13 support ea…
srowen Oct 3, 2019
854a0f7
[SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 (Part 1)
dongjoon-hyun Oct 3, 2019
4e0e4e5
[MINOR][TESTS] Rename JSONBenchmark to JsonBenchmark
dongjoon-hyun Oct 3, 2019
6b5e0e2
[SPARK-29054][SS] Invalidate Kafka consumer when new delegation token…
gaborgsomogyi Oct 3, 2019
6f68769
[SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to…
nvander1 Oct 3, 2019
aedf090
[SPARK-25468][WEBUI][FOLLOWUP] Current page index keep style with dat…
YongjinZhou Oct 3, 2019
8fabbab
[SPARK-29350] Fix BroadcastExchange reuse in Dynamic Partition Pruning
maryannxue Oct 3, 2019
0f48aaf
[SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (t…
HyukjinKwon Oct 3, 2019
8b71e54
[SPARK-29351][CORE] Avoid Full Synchronization in ShuffleMapStage
dbtsai Oct 4, 2019
91747bd
[SPARK-29326][SQL] ANSI store assignment policy: throw exception on c…
gengliangwang Oct 4, 2019
93289b5
[SPARK-29203][TESTS][MINOR][FOLLOW UP] Add access modifier for sparkC…
xuanyuanking Oct 4, 2019
275e044
[SPARK-29039][SQL] centralize the catalog and table lookup logic
cloud-fan Oct 4, 2019
eecef75
[SPARK-29355][SQL] Support timestamps subtraction
MaxGekk Oct 4, 2019
20ee2f5
[SPARK-29286][PYTHON][TESTS] Uses UTF-8 with 'replace' on errors at P…
HyukjinKwon Oct 4, 2019
130e9ae
[SPARK-29357][SQL][TESTS] Fix flaky test by changing to use AtomicLong
xuanyuanking Oct 4, 2019
228b1ea
[SPARK-28813][DOC][SQL] Document SHOW CREATE TABLE in SQL Reference
huaxingao Oct 4, 2019
df22535
[SPARK-28985][PYTHON][ML][FOLLOW-UP] Add _AFTSurvivalRegressionParams
zero323 Oct 4, 2019
8556710
[SPARK-28985][PYTHON][ML][FOLLOW-UP] Add _IsotonicRegressionBase
zero323 Oct 4, 2019
80afc79
[SPARK-29263][SCHEDULER][FOLLOWUP][TEST] Update `FakeTask.createTaskS…
jiangxb1987 Oct 5, 2019
7c5db45
[SPARK-29363][MLLIB] Make o.a.s.regression.Regressor public
zero323 Oct 6, 2019
77510c6
[SPARK-29233][K8S] Add regex expression checks for executorEnv…
merrily01 Oct 6, 2019
5a512e8
[SPARK-28800][DOC][SQL] Document REPAIR TABLE statement in SQL Reference
huaxingao Oct 6, 2019
932e261
[SPARK-29365][SQL] Support dates and timestamps subtraction
MaxGekk Oct 7, 2019
18b7ad2
[SPARK-29328][SQL] Fix calculation of mean seconds per month
MaxGekk Oct 7, 2019
ea8b5df
[SPARK-28938][K8S] Move to supported OpenJDK docker image for Kubernetes
viirya Oct 7, 2019
bd213a0
[SPARK-29360][PYTHON][ML] PySpark FPGrowthModel supports getter/setter
huaxingao Oct 7, 2019
2399134
[SPARK-29143][PYTHON][ML] Pyspark feature models support column sette…
huaxingao Oct 7, 2019
b103449
[SPARK-29342][SQL] Make casting of string values to intervals case in…
MaxGekk Oct 7, 2019
f0534fb
[SPARK-28816][DOC][SQL] Document ADD JAR statement in SQL Reference
huaxingao Oct 7, 2019
64fe82b
[SPARK-29189][SQL] Add an option to ignore block locations when listi…
wangshisan Oct 7, 2019
d2f21b0
[SPARK-27468][CORE] Track correct storage level of RDDs and partitions
Oct 7, 2019
56a3beb
[SPARK-27492][DOC][FOLLOWUP] Update resource scheduling user docs
jiangxb1987 Oct 7, 2019
cb50177
[SPARK-25668][SQL][TESTS] Refactor TPCDSQueryBenchmark to use main me…
dongjoon-hyun Oct 8, 2019
7d80aa5
[MINOR][BUILD] Fix an incorrect path in license file
beliefer Oct 8, 2019
ffddfc8
[SPARK-29269][PYTHON][ML] Pyspark ALSModel support getters/setters
huaxingao Oct 8, 2019
948a6e8
[SPARK-28892][SQL][FOLLOWUP] add resolved logical plan for UPDATE TABLE
cloud-fan Oct 8, 2019
ef1e849
[SPARK-29366][SQL] Subqueries created for DPP are not printed in EXPL…
dilipbiswal Oct 8, 2019
4e6d31f
[SPARK-24640][SQL] Return `NULL` from `size(NULL)` by default
MaxGekk Oct 8, 2019
de360e9
[SPARK-29336][SQL] Fix the implementation of QuantileSummaries.merge …
sitegui Oct 8, 2019
0da667d
[SPARK-28917][CORE] Synchronize access to RDD mutable state
squito Oct 8, 2019
b3eba29
[SPARK-29189][FOLLOW-UP][SQL] Beautify config name
wangshisan Oct 8, 2019
69b0cc1
[SPARK-28797][DOC] Document DROP FUNCTION statement in SQL Reference
sandeep-katta Oct 9, 2019
2d871ad
[SPARK-29392][CORE][SQL][STREAMING] Remove symbol literal syntax 'foo…
srowen Oct 9, 2019
ee83d09
[SPARK-29401][CORE][ML][SQL][GRAPHX][TESTS] Replace calls to .paralle…
srowen Oct 9, 2019
a927f1a
[SPARK-29373][SQL] DataSourceV2: Commands should not submit a spark job
imback82 Oct 9, 2019
c97b3ed
[SPARK-24640][SQL][FOLLOWUP] Update the SQL migration guide about `si…
MaxGekk Oct 9, 2019
ba4d413
[SPARK-29346][SQL] Add Aggregating Accumulator
hvanhovell Oct 9, 2019
fa95a5c
[SPARK-29411][CORE][ML][SQL][DSTREAM] Replace use of Unit object with…
srowen Oct 9, 2019
3b0bca4
[SPARK-29401][FOLLOWUP] Additional cases where a .parallelize call wi…
srowen Oct 9, 2019
4d93fb7
[SPARK-29413][CORE] Rewrite ThreadUtils.parmap to avoid TraversableLi…
srowen Oct 9, 2019
cc7493f
[SPARK-29416][CORE][ML][SQL][MESOS][TESTS] Use .sameElements to compa…
srowen Oct 10, 2019
7ba16ff
[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now
HyukjinKwon Oct 10, 2019
6edabeb
[SPARK-28989][SQL][FOLLOWUP] Update ANSI mode related config names in…
gengliangwang Oct 10, 2019
e946104
[SPARK-29400][CORE] Improve PrometheusResource to use labels
dongjoon-hyun Oct 10, 2019
2b3c379
[SPARK-29032][FOLLOWUP][DOCS] Add PrometheusServlet in the monitoring…
LucaCanali Oct 10, 2019
6390f02
[SPARK-29367][DOC] Add compatibility note for Arrow 0.15.0 to SQL guide
BryanCutler Oct 11, 2019
beb8d2f
[SPARK-29402][PYTHON][TESTS] Added tests for grouped map pandas_udf w…
BryanCutler Oct 11, 2019
8136295
[SPARK-29116][PYTHON][ML] Refactor py classes related to DecisionTree
huaxingao Oct 12, 2019
aa1acfe
[SPARK-28810][DOC][SQL] Document SHOW TABLES in SQL Reference
shivusondur Oct 12, 2019
3a7126c
[SPARK-29410][BUILD] Update commons-beanutils to 1.9.4
peter-toth Oct 12, 2019
8b62399
[SPARK-29380][ML] RFormula avoid repeated 'first' jobs to get vector …
zhengruifeng Oct 12, 2019
f302c2e
[SPARK-29328][SQL][FOLLOWUP] Revert calculation of mean seconds per m…
MaxGekk Oct 12, 2019
e94abd7
[SPARK-29323][WEBUI] Add tooltip for The Executors Tab's column names…
liucht-inspur Oct 12, 2019
b5b1b69
[SPARK-29445][CORE] Bump netty-all from 4.1.39.Final to 4.1.42.Final
Fokko Oct 12, 2019
782a94d
[SPARK-29433][WEBUI] Fix tooltip stages table
planga82 Oct 12, 2019
d193248
[SPARK-29368][SQL][TEST] Port interval.sql
MaxGekk Oct 13, 2019
abba53e
[SPARK-27831][FOLLOWUP][SQL][TEST] ADDITIONAL_REMOTE_REPOSITORIES is …
dongjoon-hyun Oct 13, 2019
9e12c94
[SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftSer…
peter-toth Oct 13, 2019
148cd26
[SPARK-26321][SQL] Port HIVE-15297: Hive should not split semicolon w…
wangyum Oct 13, 2019
ef81525
[SPARK-29308][BUILD] Update deps in dev/deps/spark-deps-hadoop-3.2 fo…
AngersZhuuuu Oct 13, 2019
67e1360
[SPARK-29377][PYTHON][ML] Parity between Scala ML tuning and Python M…
huaxingao Oct 14, 2019
ef6dce2
[SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
imback82 Oct 14, 2019
da576a7
[SPARK-29369][SQL] Support string intervals without the `interval` pr…
MaxGekk Oct 14, 2019
cfcaf52
[SPARK-29381][PYTHON][ML] Add _ before the XXXParams classes
huaxingao Oct 14, 2019
ba04562
[SPARK-29435][CORE] MapId in Shuffle Block is inconsistent at the wri…
sandeep-katta Oct 14, 2019
81915da
[SPARK-29107][SQL][TESTS] Port window.sql (Part 1)
DylanGuedes Oct 14, 2019
e696c36
[SPARK-29442][SQL] Set `default` mode should override the existing mode
dongjoon-hyun Oct 14, 2019
52186af
[SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
ifilonenko Oct 14, 2019
a42d894
[SPARK-29417][CORE] Resource Scheduling - add TaskContext.resource ja…
Oct 14, 2019
bfa09cf
[SPARK-29463][SQL] move v2 commands to a new file
cloud-fan Oct 14, 2019
ff9fcd5
Revert "[SPARK-29107][SQL][TESTS] Port window.sql (Part 1)"
dongjoon-hyun Oct 14, 2019
1f1443e
[SPARK-29347][SQL] Add JSON serialization for external Rows
hvanhovell Oct 14, 2019
9407fba
[SPARK-29412][SQL] refine the document of v2 session catalog config
cloud-fan Oct 15, 2019
857f109
[SPARK-10614][CORE] Add monotonic time to Clock interface
Oct 15, 2019
a988aaf
[SPARK-29454][SQL] Reduce unsafeProjection times when read Parquet fi…
LuciferYang Oct 15, 2019
8915966
[SPARK-29473][SQL] move statement logical plans to a new file
cloud-fan Oct 15, 2019
9ac4b2d
[SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader wh…
JkSelf Oct 15, 2019
2e28622
[SPARK-28211][CORE][SHUFFLE] Propose Shuffle Driver Components API
yifeih Oct 15, 2019
322ec0b
[SPARK-28885][SQL] Follow ANSI store assignment rules in table insert…
gengliangwang Oct 15, 2019
4ecbdbb
[SPARK-29182][CORE] Cache preferred locations of checkpointed RDD
viirya Oct 15, 2019
39d53d3
[SPARK-29470][BUILD] Update plugins to latest versions
dongjoon-hyun Oct 15, 2019
02c5b4f
[SPARK-28947][K8S] Status logging not happens at an interval for live…
yaooqinn Oct 15, 2019
95de93b
[SPARK-24540][SQL] Support for multiple character delimiter in Spark …
jeff303 Oct 15, 2019
51f10ed
[SPARK-28560][SQL][FOLLOWUP] code cleanup for local shuffle reader
cloud-fan Oct 16, 2019
e00344e
[SPARK-29423][SS] lazily initialize StreamingQueryManager in SessionS…
wangyum Oct 16, 2019
93e71e6
[SPARK-29469][SHUFFLE] Avoid retries by RetryingBlockFetcher when Ext…
viirya Oct 16, 2019
57edb42
[SPARK-27259][CORE] Allow setting -1 as length for FileBlock
prasha2 Oct 16, 2019
eb8c420
[SPARK-29349][SQL] Support FETCH_PRIOR in Thriftserver fetch request
juliuszsompolski Oct 16, 2019
9dacdd3
[SPARK-23578][ML][PYSPARK] Binarizer support multi-column
zhengruifeng Oct 16, 2019
5a482e7
[SPARK-29468][SQL] Change Literal.sql to be correct for floats
jose-torres Oct 16, 2019
d11cbf2
[SPARK-29364][SQL] Return an interval from date subtract according to…
MaxGekk Oct 16, 2019
6d4cc7b
[SPARK-27880][SQL] Add bool_and for every and bool_or for any as fun…
yaooqinn Oct 16, 2019
8eb8f74
[SPARK-29483][BUILD] Bump Jackson to 2.10.0
Fokko Oct 16, 2019
fda4070
[SPARK-29283][SQL] Error message is hidden when query from JDBC, espe…
LantaoJin Oct 17, 2019
239ee3f
[SPARK-9853][CORE] Optimize shuffle fetch of continuous partition IDs
xuanyuanking Oct 17, 2019
4b902d3
[SPARK-29491][SQL] Add bit_count function support
yaooqinn Oct 17, 2019
dc0bc7a
[MINOR][DOCS] Fix some typos
Oct 17, 2019
f800fa3
[SPARK-29436][K8S] Support executor for selecting scheduler through s…
merrily01 Oct 17, 2019
2f0a38c
[SPARK-29398][CORE] Support dedicated thread pools for RPC endpoints
Oct 17, 2019
100fc58
[SPARK-28869][CORE] Roll over event log files
HeartSaVioR Oct 17, 2019
78bdcfa
[SPARK-27812][K8S] Bump K8S client version to 4.6.1
igorcalabria Oct 17, 2019
00347a3
[SPARK-28762][CORE] Read JAR main class if JAR is not located in loca…
igozali Oct 17, 2019
901ff92
[SPARK-29464][PYTHON][ML] PySpark ML should expose Params.clear() to …
huaxingao Oct 18, 2019
8616109
[SPARK-9853][CORE][FOLLOW-UP] Regularize all the shuffle configuratio…
xuanyuanking Oct 18, 2019
ec5d698
[SPARK-29092][SQL] Report additional information about DataSourceScan…
dilipbiswal Oct 18, 2019
78b0cbe
[SPARK-29444] Add configuration to support JacksonGenrator to keep fi…
Oct 18, 2019
ef4c298
[SPARK-29405][SQL] Alter table / Insert statements should not change …
yaooqinn Oct 18, 2019
5692680
[SPARK-29295][SQL] Insert overwrite to Hive external table partition …
viirya Oct 18, 2019
6f8c001
[SPARK-29381][FOLLOWUP][PYTHON][ML] Add 'private' _XXXParams classes …
huaxingao Oct 18, 2019
dba673f
[SPARK-29489][ML][PYSPARK] ml.evaluation support log-loss
zhengruifeng Oct 18, 2019
7435146
[SPARK-29482][SQL] ANALYZE TABLE should look up catalog/table like v2…
cloud-fan Oct 18, 2019
39af51d
[SPARK-29014][SQL] DataSourceV2: Fix current/default catalog usage
imback82 Oct 18, 2019
c3a0d02
[SPARK-28560][SQL][FOLLOWUP] resolve the remaining comments for PR#25295
JkSelf Oct 18, 2019
77fe8a8
[SPARK-28420][SQL] Support the `INTERVAL` type in `date_part()`
MaxGekk Oct 18, 2019
9e42c52
[MINOR][DOCS] Fix incorrect EqualNullSafe symbol in sql-migration-gui…
wangyum Oct 18, 2019
9a3dcca
[SPARK-29379][SQL] SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'
AngersZhuuuu Oct 18, 2019
23f45f1
[SPARK-29515][CORE] MapStatuses SerDeser Benchmark
dbtsai Oct 18, 2019
4cfce3e
[SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting…
rahulsmahadev Oct 18, 2019
e4b4a35
[SPARK-29466][WEBUI] Show `Duration` for running drivers in Standalon…
dongjoon-hyun Oct 18, 2019
2437878
[SPARK-29502][SQL] typed interval expression should fail for invalid …
cloud-fan Oct 18, 2019
ab92e17
[SPARK-29512][SQL] REPAIR TABLE should look up catalog/table like v2 …
imback82 Oct 19, 2019
4a6005c
[SPARK-29235][ML][PYSPARK] Support avgMetrics in read/write of CrossV…
shahidki31 Oct 19, 2019
0f65b49
[SPARK-29525][SQL][TEST] Fix the associated location already exists i…
wangyum Oct 20, 2019
f4d5aa4
[SPARK-29434][CORE] Improve the MapStatuses Serialization Performance
dbtsai Oct 20, 2019
5b4d917
[SPARK-27879][SQL] Add support for bit_and and bit_or aggregates
yaooqinn Oct 21, 2019
e99a9f7
[SPARK-29498][SQL] CatalogTable to HiveTable should not change the ta…
wangyum Oct 21, 2019
5fc363b
[SPARK-29528][BUILD][TEST-MAVEN] Upgrade scala-maven-plugin to 4.2.4 …
dongjoon-hyun Oct 21, 2019
31a5dea
[SPARK-29531][SQL][TEST] refine ThriftServerQueryTestSuite.blackList …
fuwhu Oct 21, 2019
6ffec5e
[SPARK-29533][SQL][TEST] Benchmark casting strings to intervals
MaxGekk Oct 22, 2019
eef11ba
[SPARK-29518][SQL][TEST] Benchmark `date_part` for `INTERVAL`
MaxGekk Oct 22, 2019
bb4400c
[SPARK-29108][SQL][TESTS] Port window.sql (Part 2)
DylanGuedes Oct 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-29162][SQL] Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
### What changes were proposed in this pull request?
Rewrite
```
NOT isnull(x)     -> isnotnull(x)
NOT isnotnull(x)  -> isnull(x)
```

### Why are the changes needed?
Make LogicalPlan more readable and  useful for query canonicalization. Make same condition equal when judge query canonicalization equal

### Does this PR introduce any user-facing change?

NO

### How was this patch tested?

Newly added UTs.

Closes apache#25878 from AngersZhuuuu/SPARK-29162.

Authored-by: angerszhu <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
AngersZhuuuu authored and dongjoon-hyun committed Sep 22, 2019
commit fe4bee8fd825f8b123569ded6e50e567f5cf9e2c
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,9 @@ object BooleanSimplification extends Rule[LogicalPlan] with PredicateHelper {
case Not(a And b) => Or(Not(a), Not(b))

case Not(Not(e)) => e

case Not(IsNull(e)) => IsNotNull(e)
case Not(IsNotNull(e)) => IsNull(e)
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,11 @@ class BooleanSimplificationSuite extends PlanTest with ExpressionEvalHelper with
checkCondition(!'f || 'e, testRelationWithData.where(!'f || 'e).analyze)
}

test("simplify NOT(IsNull(x)) and NOT(IsNotNull(x))") {
checkCondition(Not(IsNotNull('b)), IsNull('b))
checkCondition(Not(IsNull('b)), IsNotNull('b))
}

protected def assertEquivalent(e1: Expression, e2: Expression): Unit = {
val correctAnswer = Project(Alias(e2, "out")() :: Nil, OneRowRelation()).analyze
val actual = Optimize.execute(Project(Alias(e1, "out")() :: Nil, OneRowRelation()).analyze)
Expand Down