AE_1#3

Open

markhamstra wants to merge 2462 commits intomarkhamstra:AEfrom

carsonwang:AE_1

This pull request is big! We're only showing the most recent 250 commits

Commits on Dec 4, 2018

[MINOR][SQL] Combine the same codes in test cases

彭灿00244106
authored and
maropu
committed
[SPARK-24423][FOLLOW-UP][SQL] Fix error example

wangyum
authored and
srowen
committed
[SPARK-26178][SQL] Use java.time API for parsing timestamps and dates from CSV

MaxGekk
authored and
srowen
committed
[SPARK-26233][SQL] CheckOverflow when encoding a decimal value

mgaido91
authored and
dongjoon-hyun
committed
[SPARK-26119][CORE][WEBUI] Task summary table should contain only successful tasks' metrics

shahidki31
authored and
Marcelo Vanzin
committed
[SPARK-26094][CORE][STREAMING] createNonEcFile creates parent dirs.

squito
authored and
Marcelo Vanzin
committed

Commits on Dec 5, 2018

Commits on Dec 6, 2018

[SPARK-26275][PYTHON][ML] Increases timeout for StreamingLogisticRegressionWithSGDTests.test_training_and_prediction test
HyukjinKwon
committed
[SPARK-25274][PYTHON][SQL] In toPandas with Arrow send un-ordered record batches to improve performance
BryanCutler
committed
[SPARK-26236][SS] Add kafka delegation token support documentation.

gaborgsomogyi
authored and
Marcelo Vanzin
committed
[SPARK-26194][K8S] Auto generate auth secret for k8s apps.

Marcelo Vanzin
authored and
mccheah
committed

Commits on Dec 7, 2018

[SPARK-26289][CORE] cleanup enablePerfMetrics parameter from BytesToBytesMap

heary-cao
authored and
cloud-fan
committed
[SPARK-26263][SQL] Validate partition values with user provided schema

gengliangwang
authored and
cloud-fan
committed
[SPARK-26298][BUILD] Upgrade Janino to 3.0.11
dongjoon-hyun
committed
[SPARK-26060][SQL][FOLLOW-UP] Rename the config name.

ueshin
authored and
dongjoon-hyun
committed
[SPARK-24243][CORE] Expose exceptions from InProcessAppHandle

Sahil Takiar
authored and
Marcelo Vanzin
committed
[SPARK-26294][CORE] Delete Unnecessary If statement

wangjiaochun
authored and
srowen
committed
[MINOR][SQL][DOC] Correct parquet nullability documentation

dima-asana
authored and
srowen
committed
[SPARK-26196][SPARK-26281][WEBUI] Total tasks title in the stage page is incorrect when there are failed or killed tasks and update duration metrics

shahidki31
authored and
srowen
committed
[SPARK-24333][ML][PYTHON] Add fit with validation set to spark.ml GBT: Python API

huaxingao
authored and
BryanCutler
committed
[SPARK-26304][SS] Add default value to spark.kafka.sasl.kerberos.service.name parameter

gaborgsomogyi
authored and
Marcelo Vanzin
committed

Commits on Dec 8, 2018

Commits on Dec 9, 2018

Commits on Dec 10, 2018

[SPARK-26287][CORE] Don't need to create an empty spill file when memory has no records

wangjiaochun
authored and
dongjoon-hyun
committed
[SPARK-26307][SQL] Fix CTAS when INSERT a partitioned table using Hive serde

gatorsmile
authored and
cloud-fan
committed
[SPARK-26319][SQL][TEST] Add appendReadColumns Unit Test for HiveShimSuite

da-liii
authored and
HyukjinKwon
committed
[SPARK-26286][TEST] Add MAXIMUM_PAGE_SIZE_BYTES exception bound unit test

wangjiaochun
authored and
HyukjinKwon
committed
[MINOR][DOC] Update the condition description of serialized shuffle

10110346
authored and
cloud-fan
committed
This tests pushing to gitbox
srowen
committed
This tests pushing directly to github
srowen
committed
[SPARK-24958][CORE] Add memory from procfs to executor metrics.

Reza Safi
authored and
squito
committed
[SPARK-26317][BUILD] Upgrade SBT to 0.13.18
dongjoon-hyun
committed

Commits on Dec 11, 2018

[SPARK-25696] The storage memory displayed on spark Application UI is…

httfighter
authored and
srowen
committed
[SPARK-19827][R] spark.ml R API for PIC

huaxingao
authored and
srowen
committed
[SPARK-26312][SQL] Replace RDDConversions.rowToRowRdd with RowEncoder to improve its conversion performance

eatoncys
authored and
HyukjinKwon
committed
[SPARK-26293][SQL] Cast exception when having python udf in subquery
cloud-fan
committed
[SPARK-26303][SQL] Return partial results for bad JSON records

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-26327][SQL] Bug fix for `FileSourceScanExec` metrics update and name changing

xuanyuanking
authored and
cloud-fan
committed
[SPARK-26265][CORE] Fix deadlock in BytesToBytesMap.MapIterator when locking both BytesToBytesMap.MapIterator and TaskMemoryManager

viirya
authored and
cloud-fan
committed
[SPARK-26316][SPARK-21052] Revert hash join metrics in that causes performance degradation

JkSelf
authored and
cloud-fan
committed
[SPARK-26300][SS] Remove a redundant `checkForStreaming` call

10110346
authored and
dongjoon-hyun
committed
[SPARK-26239] File-based secret key loading for SASL.

mccheah
authored and
Marcelo Vanzin
committed

Commits on Dec 12, 2018

[SPARK-26193][SQL][FOLLOW UP] Read metrics rename and display text changes

xuanyuanking
authored and
cloud-fan
committed
[SPARK-19827][R][FOLLOWUP] spark.ml R API for PIC
srowen
committed
[SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator

imatiach-msft
authored and
srowen
committed
[SPARK-25877][K8S] Move all feature logic to feature classes.

Marcelo Vanzin
authored and
mccheah
committed

Commits on Dec 13, 2018

[SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics

LucaCanali
authored and
Marcelo Vanzin
committed
[SPARK-26322][SS] Add spark.kafka.sasl.token.mechanism to ease delegation token configuration.

gaborgsomogyi
authored and
Marcelo Vanzin
committed
[SPARK-26297][SQL] improve the doc of Distribution/Partitioning
cloud-fan
committed
[SPARK-26348][SQL][TEST] make sure expression is resolved during test
cloud-fan
committed
[SPARK-26355][PYSPARK] Add a workaround for PyArrow 0.11.

ueshin
authored and
HyukjinKwon
committed
[MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell

AzureQ
authored and
HyukjinKwon
committed
[MINOR][DOC] Fix comments of ConvertToLocalRelation rule

seancxmao
authored and
srowen
committed
[MINOR][DOC] update the condition description of BypassMergeSortShuffle

lcqzte10192193
authored and
srowen
committed
[SPARK-26340][CORE] Ensure cores per executor is greater than cpu per task

n.fraison
authored and
srowen
committed
[SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits

cloud-fan
authored and
HyukjinKwon
committed
[SPARK-26098][WEBUI] Show associated SQL query in Job page

gengliangwang
authored and
gatorsmile
committed

Commits on Dec 14, 2018

[SPARK-23886][SS] Update query status for ContinuousExecution

gaborgsomogyi
authored and
Marcelo Vanzin
committed
[SPARK-26364][PYTHON][TESTING] Clean up imports in test_pandas_udf*

icexelloss
authored and
HyukjinKwon
committed
[SPARK-26360] remove redundant validateQuery call

JasonWayne
authored and
HyukjinKwon
committed
[SPARK-26337][SQL][TEST] Add benchmark for LongToUnsafeRowMap

viirya
authored and
cloud-fan
committed
[SPARK-26368][SQL] Make it clear that getOrInferFileFormatSchema doesn't create InMemoryFileIndex

rxin
authored and
gatorsmile
committed
[SPARK-26370][SQL] Fix resolution of higher-order function for the same identifier.

ueshin
authored and
cloud-fan
committed
[MINOR][SQL] Some errors in the notes.

CarolinPeng
authored and
srowen
committed

Commits on Dec 15, 2018

Commits on Dec 16, 2018

[SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON

MaxGekk
authored and
cloud-fan
committed
[SPARK-26078][SQL] Dedup self-join attributes on IN subqueries

mgaido91
authored and
cloud-fan
committed
[SPARK-26372][SQL] Don't reuse value from previous row when parsing bad CSV input field

bersprockets
authored and
HyukjinKwon
committed

Commits on Dec 17, 2018

[SPARK-26248][SQL] Infer date type from CSV

MaxGekk
authored and
cloud-fan
committed
[MINOR][DOCS] Fix the "not found: value Row" error on the "programmatic_schema" example

kjmrknsn
authored and
dongjoon-hyun
committed
Revert "[SPARK-26248][SQL] Infer date type from CSV"
HyukjinKwon
committed
[SPARK-26352][SQL] join reorder should not change the order of output attributes

rednaxelafx
authored and
cloud-fan
committed
[SPARK-26327][SQL][FOLLOW-UP] Refactor the code and restore the metrics name
gatorsmile
committed
[SPARK-20636] Add the rule TransposeWindow to the optimization batch
gatorsmile
committed
[SPARK-26243][SQL][FOLLOWUP] fix code style issues in TimestampFormatter.scala

cloud-fan
authored and
HyukjinKwon
committed
[SPARK-20351][ML] Add trait hasTrainingSummary to replace the duplicate code

YY-OnCall
authored and
srowen
committed
[SPARK-26255][YARN] Apply user provided UI filters to SQL tab in yarn mode

chakravarthiT
authored and
Marcelo Vanzin
committed
[SPARK-26371][SS] Increase kafka ConfigUpdater test coverage.

gaborgsomogyi
authored and
dongjoon-hyun
committed
[SPARK-24933][SS] Report numOutputRows in SinkProgress

authored and
gatorsmile
committed
[SPARK-25922][K8] Spark Driver/Executor "spark-app-selector" label mismatch

authored and
liyinan926
committed

Commits on Dec 18, 2018

[SPARK-24561][SQL][PYTHON] User-defined window aggregation functions with Pandas UDF (bounded window)

icexelloss
authored and
HyukjinKwon
committed
[SPARK-26246][SQL] Inferring TimestampType from JSON

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-26081][SQL][FOLLOW-UP] Use foreach instead of misuse of map (for Unit)
HyukjinKwon
committed
[SPARK-24680][DEPLOY] Support spark.executorEnv.JAVA_HOME in Standalone mode

stanzhai
authored and
srowen
committed
[SPARK-26384][SQL] Propagate SQL configs for CSV schema inferring

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-26382][CORE] prefix comparator should handle -0.0

cloud-fan
authored and
dongjoon-hyun
committed
[SPARK-26394][CORE] Fix annotation error for Utils.timeStringAsMs

Jackey Lee
authored and
srowen
committed
[SPARK-25815][K8S] Support kerberos in client mode, keytab-based token renewal.
Marcelo Vanzin
committed

Commits on Dec 19, 2018

[SPARK-26366][SQL] ReplaceExceptWithFilter should consider NULL as False

mgaido91
authored and
gatorsmile
committed
[SPARK-26390][SQL] ColumnPruning rule should only do column pruning

cloud-fan
authored and
dongjoon-hyun
committed

Commits on Dec 20, 2018

[SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE

maropu
authored and
cloud-fan
committed
[SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible

viirya
authored and
cloud-fan
committed
[SPARK-26318][SQL] Deprecate Row.merge

KyleLi1985
authored and
HyukjinKwon
committed
[SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF

mgaido91
authored and
cloud-fan
committed
[SPARK-24687][CORE] Avoid job hanging when generate task binary causes fatal error

caneGuy
authored and
srowen
committed
[SPARK-26324][DOCS] Add Spark docs for Running in Mesos with SSL

authored and
srowen
committed
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions

gengliangwang
authored and
gatorsmile
committed
[SPARK-26392][YARN] Cancel pending allocate requests by taking locality preference into account

Ngone51
authored and
Marcelo Vanzin
committed
[SPARK-25970][ML] Add Instrumentation to PrefixSpan

zhengruifeng
authored and
mengxr
committed

Commits on Dec 21, 2018

Commits on Dec 22, 2018

Commits on Dec 24, 2018

Commits on Dec 25, 2018

[SPARK-26426][SQL] fix ExpresionInfo assert error in windows operation system.

wangyanlin01
authored and
HyukjinKwon
committed

Commits on Dec 27, 2018

[SPARK-26424][SQL] Use java.time API in date/timestamp expressions

MaxGekk
authored and
cloud-fan
committed
[SPARK-26435][SQL] Support creating partitioned table using Hive CTAS by specifying partition column names

viirya
authored and
cloud-fan
committed
[SPARK-26191][SQL] Control truncation of Spark plans via maxFields parameter

MaxGekk
authored and
hvanhovell
committed
[SPARK-25892][SQL] Change AttributeReference.withMetadata's return type to AttributeReference

kevinyu98
authored and
HyukjinKwon
committed
[SPARK-26451][SQL] Change lead/lag argument name from count to offset

deepyaman
authored and
HyukjinKwon
committed

Commits on Dec 28, 2018

[SPARK-26446][CORE] Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

wuqingxin
authored and
dongjoon-hyun
committed
[SPARK-26444][WEBUI] Stage color doesn't change with it's status

seancxmao
authored and
srowen
committed
[SPARK-26424][SQL][FOLLOWUP] Fix DateFormatClass/UnixTime codegen
dongjoon-hyun
committed

Commits on Dec 29, 2018

[SPARK-26496][SS][TEST] Avoid to use Random.nextString in StreamingInnerJoinSuite

HyukjinKwon
authored and
dongjoon-hyun
committed

Commits on Dec 30, 2018

[SPARK-26443][CORE] Use ConfigEntry for hardcoded configs for history category.

ueshin
authored and
dongjoon-hyun
committed
[SPARK-26363][WEBUI] Avoid duplicated KV store lookups in method `taskList`

gengliangwang
authored and
srowen
committed

Commits on Dec 31, 2018

Commits on Jan 1, 2019

[MINOR] Fix inconsistency log level among delegation token providers

HeartSaVioR
authored and
HyukjinKwon
committed
Revert "[SPARK-26339][SQL] Throws better exception when reading files that start with underscore"
HyukjinKwon
committed
[SPARK-26499][SQL] JdbcUtils.makeGetter does not handle ByteType

twdsilva
authored and
HyukjinKwon
committed
[SPARK-24544][SQL] Print actual failure cause when look up function failed

caneGuy
authored and
srowen
committed
[SPARK-25765][ML] Add training cost to BisectingKMeans summary

mgaido91
authored and
srowen
committed
[SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest

MaxGekk
authored and
HyukjinKwon
committed

Commits on Jan 3, 2019

[SPARK-26023][SQL][FOLLOWUP] Dumping truncated plans and generated code to a file

MaxGekk
authored and
gatorsmile
committed
[SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon
committed
[SPARK-26502][SQL] Move hiveResultString() from QueryExecution to HiveResult

MaxGekk
authored and
hvanhovell
committed
[SPARK-26501][CORE][TEST] Fix unexpected overriden of exitFn in SparkSubmitSuite

Liupengcheng
authored and
srowen
committed
[SPARK-26517][SQL][TEST] Avoid duplicate test in ParquetSchemaPruningSuite

viirya
authored and
srowen
committed
[SPARK-26447][SQL] Allow OrcColumnarBatchReader to return less partition columns

authored and
cloud-fan
committed
[SPARK-26489][CORE] Use ConfigEntry for hardcoded configs for python/r categories

HeartSaVioR
authored and
Marcelo Vanzin
committed

Commits on Jan 4, 2019

Commits on Jan 5, 2019

[SPARK-26537][BUILD] change git-wip-us to gitbox

shaneknapp
authored and
dongjoon-hyun
committed
[SPARK-26536][BUILD][TEST] Upgrade Mockito to 2.23.4
dongjoon-hyun
committed
[SPARK-26541][BUILD] Add `-Pdocker-integration-tests` to `dev/scalastyle`
dongjoon-hyun
committed
[SPARK-26078][SQL][FOLLOWUP] Remove useless import

mgaido91
authored and
dongjoon-hyun
committed
[SPARK-26246][SQL][FOLLOWUP] Inferring TimestampType from JSON

MaxGekk
authored and
HyukjinKwon
committed
[MINOR][DOC] Fix typos in the SQL migration guide

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-26545] Fix typo in EqualNullSafe's truth table comment

rednaxelafx
authored and
gatorsmile
committed

Commits on Jan 6, 2019

[SPARK-26548][SQL] Don't hold CacheManager write lock while computing executedPlan

DaveDeCaprio
authored and
gatorsmile
committed
[SPARK-26527][CORE] Let acquireUnrollMemory fail fast if required space exceeds memory limit

SongYadong
authored and
srowen
committed
[SPARK-26339][SQL] Throws better exception when reading files that start with underscore

Hirobe Keiichi
authored and
srowen
committed

Commits on Jan 7, 2019

[SPARK-26547][SQL] Remove duplicate toHiveString from HiveUtils

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-26554][BUILD] Update `release-util.sh` to avoid GitBox fake 200 headers
dongjoon-hyun
committed
[SPARK-26536][BUILD][FOLLOWUP][TEST-MAVEN] Make StreamingReadSupport public for maven testing
dongjoon-hyun
committed
[MINOR][BUILD] Fix script name in `release-tag.sh` usage message
dongjoon-hyun
committed
[SPARK-26559][ML][PYSPARK] ML image can't work with numpy versions prior to 1.9

viirya
authored and
HyukjinKwon
committed
[SPARK-26383][CORE] NPE when use DataFrameReader.jdbc with wrong URL

ayudovin
authored and
srowen
committed
[SPARK-24489][ML] Check for invalid input type of weight data in ml.PowerIterationClustering

shahidki31
authored and
holdenk
committed
[SPARK-25689][YARN] Make driver, not AM, manage delegation tokens.

Marcelo Vanzin
authored and
squito
committed
[SPARK-26065][SQL] Change query hint from a `LogicalPlan` to a field

maryannxue
authored and
gatorsmile
committed
[SPARK-26491][CORE][TEST] Use ConfigEntry for hardcoded configs for test categories

mgaido91
authored and
Marcelo Vanzin
committed
[SPARK-26339][SQL][FOLLOW-UP] Issue warning instead of throwing an exception for underscore files

HyukjinKwon
authored and
gatorsmile
committed

Commits on Jan 8, 2019

[MINOR][K8S] add missing docs for podTemplateContainerName properties

aditanase
authored and
srowen
committed
[SPARK-26554][BUILD][FOLLOWUP] Use GitHub instead of GitBox to check HEADER
dongjoon-hyun
committed
[SPARK-24196][SQL] Implement Spark's own GetSchemasOperation

wangyum
authored and
gatorsmile
committed
[SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any

cloud-fan
authored and
HyukjinKwon
committed
[MINOR][WEBUI] Modify the name of the column named "shuffle spill" in the StagePage

10110346
authored and
srowen
committed
[SPARK-26002][SQL] Fix day of year calculation for Julian calendar days

attilapiros
authored and
cloud-fan
committed
[SPARK-24522][UI] Create filter to apply HTTP security checks consistently.

Marcelo Vanzin
authored and
squito
committed
[SPARK-24920][CORE] Allow sharing Netty's memory pool allocators

attilapiros
authored and
srowen
committed
[SPARK-26349][PYSPARK] Forbid insecure py4j gateways

squito
authored and
BryanCutler
committed

Commits on Jan 9, 2019

[SPARK-26571][SQL] Update Hive Serde mapping with canonical name of Parquet and Orc FileFormat

gengliangwang
authored and
cloud-fan
committed
[SPARK-26529] Add debug logs for confArchive when preparing local resource

Liupengcheng
authored and
HyukjinKwon
committed
[SPARK-26549][PYSPARK] Fix for python worker reuse take no effect for parallelize lazy iterable range

xuanyuanking
authored and
HyukjinKwon
committed
[SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

committed
[SPARK-26448][SQL] retain the difference between 0.0 and -0.0

cloud-fan
authored and
gatorsmile
committed
[SPARK-26065][FOLLOW-UP][SQL] Fix the Failure when having two Consecutive Hints

maryannxue
authored and
gatorsmile
committed

Commits on Jan 10, 2019

Commits on Jan 11, 2019

Commits on Jan 13, 2019

[SPARK-26503][CORE][DOC][FOLLOWUP] Get rid of spark.sql.legacy.timeParser.enabled

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-23182][CORE] Allow enabling TCP keep alive on the RPC connections

peshopetrov
authored and
srowen
committed
[SPARK-26450][SQL] Avoid rebuilding map of schema for every column in projection

bersprockets
authored and
hvanhovell
committed
[SPARK-26065][FOLLOW-UP][SQL] Revert hint behavior in join reordering

maryannxue
authored and
gatorsmile
committed
[SPARK-26576][SQL] Broadcast hint not applied to partitioned table

jzhuge
authored and
gatorsmile
committed

Commits on Jan 14, 2019

Commits on Jan 15, 2019

Commits on Jan 22, 2019

simplify QueryStage (#5)

cloud-fan
authored and
carsonwang
committed

Commits on Jan 31, 2019

insert query stages dynamically (#6)

cloud-fan
authored and
carsonwang
committed

Commits on Feb 1, 2019

address comments
carsonwang
committed

Commits on Feb 14, 2019

Fix UT (#7)

xuanyuanking
authored and
carsonwang
committed

Commits on Feb 25, 2019

rename to QueryFragment
carsonwang
committed

Commits on Feb 27, 2019

fix test
carsonwang
committed

Commits on Feb 28, 2019

Commits on Mar 15, 2019

improve (#9)

cloud-fan
authored and
carsonwang
committed
rename
carsonwang
committed

Commits on Mar 22, 2019

address comments
carsonwang
committed