Ky 945, sync from apache spark#401

Merged

fishcus merged 169 commits intoKyligence:kyspark-3.1.1.x-4.x-xuanwufrom

Feb 16, 2022

Commits on Jan 10, 2022

[SPARK-34361][K8S] In case of downscaling avoid killing of executors already known by the scheduler backend in the pod allocator

attilapiros
authored and
fishcus
committed
[SPARK-34948][K8S] Add ownerReference to executor configmap to fix leakages

dongjoon-hyun
authored and
fishcus
committed
[SPARK-35482][K8S] Use `spark.blockManager.port` not the wrong `spark.blockmanager.port` in BasicExecutorFeatureStep

yaooqinn
authored and
fishcus
committed
[SPARK-35493][K8S] make `spark.blockManager.port` fallback for `spark.driver.blockManager.port` as same as other cluster managers

yaooqinn
authored and
fishcus
committed
[SPARK-32975][K8S] Add config for driver readiness timeout before executors start

authored and
fishcus
committed
[SPARK-32975][K8S][FOLLOWUP] Avoid None.get exception

yaooqinn
authored and
fishcus
committed
[MINOR][K8S] Print the driver pod name instead of Some(name) if absent

yaooqinn
authored and
fishcus
committed
[SPARK-37049][K8S] executorIdleTimeout should check `creationTimestamp` instead of `startTime`

yangwwei
authored and
fishcus
committed

Commits on Jan 11, 2022

[SPARK-37497][K8S] Promote `ExecutorPods[PollingSnapshot|WatchSnapshot]Source` to DeveloperApi

dongjoon-hyun
authored and
fishcus
committed
[SPARK-34949][CORE] Prevent BlockManager reregister when Executor is shutting down

sumeetgajjar
authored and
fishcus
committed
[SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

kotlovs
authored and
fishcus
committed
[SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

dongjoon-hyun
authored and
fishcus
committed
[SPARK-35879][CORE][SHUFFLE] Fix performance regression caused by collectFetchRequests

yaooqinn
authored and
fishcus
committed

Commits on Jan 12, 2022

[SPARK-34473][SQL] Avoid NPE in DataFrameReader.schema(StructType)

cloud-fan
authored and
fishcus
committed
[SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

linhongliu-db
authored and
fishcus
committed
[SPARK-34515][SQL] Fix NPE if InSet contains null value during getPartitionsByFilter

ulysses-you
authored and
fishcus
committed
[SPARK-34436][SQL] DPP support LIKE ANY/ALL expression

wangyum
authored and
fishcus
committed
[SPARK-34550][SQL] Skip InSet null value during push filter to Hive metastore

ulysses-you
authored and
fishcus
committed
[SPARK-34556][SQL] Checking duplicate static partition columns should respect case sensitive conf

zsxwing
authored and
fishcus
committed
[SPARK-34417][SQL] org.apache.spark.sql.DataFrameNaFunctions.fillMap fails for column name having a dot

amandeep-sharma
authored and
fishcus
committed
[SPARK-34547][SQL] Only use metadata columns for resolution as last resort

karenfeng
authored and
fishcus
committed
[SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode

rednaxelafx
authored and
fishcus
committed
[SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

authored and
fishcus
committed
[SPARK-34555][SQL] Resolve metadata output from DataFrame

karenfeng
authored and
fishcus
committed
[SPARK-34584][SQL] Static partition should also follow StoreAssignmentPolicy when insert into v2 tables

cloud-fan
authored and
fishcus
committed
[SPARK-34599][SQL] Fix the issue that INSERT INTO OVERWRITE doesn't support partition columns containing dot for DSv2

zsxwing
authored and
fishcus
committed
[SPARK-34567][SQL] CreateTableAsSelect should update metrics too

AngersZhuuuu
authored and
fishcus
committed
[SPARK-34482][SS] Correct the active SparkSession for StreamExecution.logicalPlan

Ngone51
authored and
fishcus
committed
[SPARK-32924][WEBUI] Make duration column in master UI sorted in the correct order

Baohe Zhang
authored and
fishcus
committed
[SPARK-34613][SQL] Fix view does not capture disable hint config

ulysses-you
authored and
fishcus
committed
[SPARK-34607][SQL][3.1] Add `Utils.isMemberClass` to fix a malformed class name error on jdk8u

maropu
authored and
fishcus
committed
[SPARK-34681][SQL] Fix bug for full outer shuffled hash join when building left side with non-equal condition

c21
authored and
fishcus
committed
[SPARK-34682][SQL] Fix regression in canonicalization error check in CustomShuffleReaderExec

andygrove
authored and
fishcus
committed
[SPARK-34682][SQL] Use PrivateMethodTester instead of reflection

andygrove
authored and
fishcus
committed
[MINOR][SQL] Remove unnecessary extend from BroadcastHashJoinExec

c21
authored and
fishcus
committed
[SPARK-34713][SQL] Fix group by CreateStruct with ExtractValue

cloud-fan
authored and
fishcus
committed
[SPARK-34724][SQL] Fix Interpreted evaluation by using getMethod instead of getDeclaredMethod

dongjoon-hyun
authored and
fishcus
committed
[SPARK-34723][SQL] Correct parameter type for subexpression elimination under whole-stage

viirya
authored and
fishcus
committed
[SPARK-34727][SQL] Fix discrepancy in casting float to timestamp

MaxGekk
authored and
fishcus
committed
[SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

cloud-fan
authored and
fishcus
committed
[SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

cloud-fan
authored and
fishcus
committed
[SPARK-34749][SQL][3.1] Simplify ResolveCreateNamedStruct

cloud-fan
authored and
fishcus
committed
[SPARK-34737][SQL][3.1] Cast input float to double in `TIMESTAMP_SECONDS`

MaxGekk
authored and
fishcus
committed
[SPARK-34731][CORE] Avoid ConcurrentModificationException when redacting properties in EventLoggingListener

bersprockets
authored and
fishcus
committed
[SPARK-34776][SQL] Nested column pruning should not prune Window produced attributes

viirya
authored and
fishcus
committed
[SPARK-34796][SQL][3.1] Initialize counter variable for LIMIT code-gen in doProduce()

c21
authored and
fishcus
committed
[SPARK-34811][CORE] Redact fs.s3a.access.key like secret and token

dongjoon-hyun
authored and
fishcus
committed
[SPARK-34225][CORE] Don't encode further when a URI form string is passed to addFile or addJar

sarutak
authored and
fishcus
committed
[SPARK-34790][CORE] Disable fetching shuffle blocks in batch when io encryption is enabled

hezuojiao
authored and
fishcus
committed
[SPARK-33482][SPARK-34756][SQL] Fix FileScan equality check

peter-toth
authored and
fishcus
committed
[SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

maropu
authored and
fishcus
committed
[SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

authored and
fishcus
committed
[SPARK-34833][SQL][FOLLOWUP] Handle outer references in all the places

cloud-fan
authored and
fishcus
committed
[SPARK-34829][SQL] Fix higher order function results

peter-toth
authored and
fishcus
committed
[SPARK-34876][SQL] Fill defaultResult of non-nullable aggregates

tanelk
authored and
fishcus
committed
[SPARK-34814][SQL] LikeSimplification should handle NULL

AngersZhuuuu
authored and
fishcus
committed
[SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

Baohe Zhang
authored and
fishcus
committed
[SPARK-34909][SQL] Fix conversion of negative to unsigned in conv()

timarmstrong
authored and
fishcus
committed
[SPARK-34939][CORE] Throw fetch failure exception when unable to deserialize broadcasted map statuses

viirya
authored and
fishcus
committed
[SPARK-34923][SQL] Metadata output should be empty for more plans

karenfeng
authored and
fishcus
committed
[SPARK-34922][SQL][3.1] Use a relative cost comparison function in the CBO

tanelk
authored and
fishcus
committed
[SPARK-34963][SQL] Fix nested column pruning for extracting case-insensitive struct field from array of struct

viirya
authored and
fishcus
committed
[SPARK-34926][SQL][3.1] PartitioningUtils.getPathFragment() should respect partition value is null

AngersZhuuuu
authored and
fishcus
committed
[SPARK-35014] Fix the PhysicalAggregation pattern to not rewrite foldable expressions

sigmod
authored and
fishcus
committed
[SPARK-34834][NETWORK] Fix a potential Netty memory leak in TransportResponseHandler

weixiuli
authored and
fishcus
committed
[SPARK-34225][CORE][FOLLOWUP] Replace Hadoop's Path with Utils.resolveURI to make the way to get URI simple

sarutak
authored and
fishcus
committed
[SPARK-35136] Remove initial null value of LiveStage.info

sander-goos
authored and
fishcus
committed
[SPARK-35117][UI] Change progress bar back to highlight ratio of tasks in progress

Kimahriman
authored and
fishcus
committed
[SPARK-35080][SQL] Only allow a subset of correlated equality predicates when a subquery is aggregated

authored and
fishcus
committed
[SPARK-34639][SQL][3.1] RelationalGroupedDataset.alias should not create UnresolvedAlias

cloud-fan
authored and
fishcus
committed
[SPARK-35096][SQL] SchemaPruning should adhere spark.sql.caseSensitive config

sandeep-katta
authored and
fishcus
committed
[SPARK-35127][UI] When we switch between different stage-detail pages, the entry item in the newly-opened page may be blank

echohlne
authored and
fishcus
committed
[SPARK-34897][SQL][3.1] Support reconcile schemas based on index after nested column pruning

wangyum
authored and
fishcus
committed
[SPARK-35168][SQL] mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

yaooqinn
authored and
fishcus
committed
[SPARK-35087][UI] Some columns in table Aggregated Metrics by Executor of stage-detail page shows incorrectly.

echohlne
authored and
fishcus
committed
[SPARK-35213][SQL] Keep the correct ordering of nested structs in chained withField operations

Kimahriman
authored and
fishcus
committed
[SPARK-35244][SQL] Invoke should throw the original exception

cloud-fan
authored and
fishcus
committed
[SPARK-35278][SQL] Invoke should find the method with correct number of parameters

viirya
authored and
fishcus
committed
[SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions

authored and
fishcus
committed
[SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match

viirya
authored and
fishcus
committed
[SPARK-35359][SQL] Insert data with char/varchar datatype will fail when data length exceed length limitation

fhygh
authored and
fishcus
committed
[SPARK-35411][SQL] Add essential information while serializing TreeNode to json

ivoson
authored and
fishcus
committed
[SPARK-35106][CORE][SQL] Avoid failing rename caused by destination directory not exist

Yuzhou Sun
authored and
fishcus
committed
[SPARK-35093][SQL] AQE now uses newQueryStage plan as key for looking up cached exchanges for re-use

andygrove
authored and
fishcus
committed
[SPARK-35287][SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation

sarutak
authored and
fishcus
committed
[SPARK-35449][SQL][3.1] Only extract common expressions from CaseWhen values if elseValue is set

Kimahriman
authored and
fishcus
committed
[SPARK-35454][SQL][3.1] One LogicalPlan can match multiple dataset ids

Ngone51
authored and
fishcus
committed
[SPARK-35411][SQL][FOLLOWUP] Handle Currying Product while serializing TreeNode to JSON

ivoson
authored and
fishcus
committed
[SPARK-35610][CORE] Fix the memory leak introduced by the Executor's stop shutdown hook

attilapiros
authored and
fishcus
committed
[SPARK-35589][CORE][3.1] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

dongjoon-hyun
authored and
fishcus
committed
[SPARK-35679][SQL] instantToMicros overflow

dchvn
authored and
fishcus
committed
[SPARK-35673][SQL] Fix user-defined hint and unrecognized hint in subquery

cfmcgrady
authored and
fishcus
committed
[SPARK-35653][SQL] Fix CatalystToExternalMap interpreted path fails for Map with case classes as keys or values

Emil Ejbyfeldt
authored and
fishcus
committed
[SPARK-35296][SQL] Allow Dataset.observe to work even if CollectMetricsExec in a task handles multiple partitions

sarutak
authored and
fishcus
committed
[SPARK-35652][SQL] joinWith on two table generated from same one

dchvn
authored and
fishcus
committed
[SPARK-35695][SQL] Collect observed metrics from cached and adaptive execution sub-trees

tanelk
authored and
fishcus
committed
[SPARK-35746][UI] Fix taskid in the stage page task event timeline

shahidki31
authored and
fishcus
committed
[SPARK-35714][CORE] Bug fix for deadlock during the executor shutdown

wankunde
authored and
fishcus
committed
[SPARK-35767][SQL] Avoid executing child plan twice in CoalesceExec

andygrove
authored and
fishcus
committed
[SPARK-35791][SQL] Release on-going map properly for NULL-aware ANTI join

c21
authored and
fishcus
committed
[SPARK-34766][SQL][3.1] Do not capture maven config for views

ulysses-you
authored and
fishcus
committed
[SPARK-35792][SQL] View should not capture configs used in `RelationConversions`

linhongliu-db
authored and
fishcus
committed
[SPARK-35391] Fix memory leak in ExecutorAllocationListener

VasilyKolpakov
authored and
fishcus
committed
[SPARK-35695][SQL][FOLLOWUP] Use AQE helper to simplify the code in CollectMetricsExec

cloud-fan
authored and
fishcus
committed
[SPARK-35841][SQL] Casting string to decimal type doesn't work if the…

dchvn
authored and
fishcus
committed
[SPARK-35886][SQL][3.1] PromotePrecision should not overwrite genCodePromotePrecision should not overwrite genCode

viirya
authored and
fishcus
committed
[SPARK-35898][SQL] Fix arrays and maps in RowToColumnConverter

tomvanbussel
authored and
fishcus
committed
[SPARK-35935][SQL][3.1][3.0] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

MaxGekk
authored and
fishcus
committed
[SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

Ngone51
authored and
fishcus
committed
[SPARK-36020][SQL][3.1] Check logical link in remove redundant projects

authored and
fishcus
committed
[SQL][MINOR] EquivalentExpressions.commonChildrenToRecurse should skip CodegenFallback

cloud-fan
authored and
fishcus
committed
[SPARK-36076][SQL][3.1] ArrayIndexOutOfBounds in Cast string to times…

dchvn
authored and
fishcus
committed
[SPARK-36034][SQL][3.1] Rebase datetime in pushed down filters to parquet

MaxGekk
authored and
fishcus
committed
[SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36093][SQL][3.1] RemoveRedundantAliases should not change Command's parameter's expression's name

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36079][SQL] Null-based filter estimate should always be in the range [0, 1]

karenfeng
authored and
fishcus
committed
[SPARK-36210][SQL] Preserve column insertion order in Dataset.withColumns

koertkuipers
authored and
fishcus
committed
[SPARK-35027][CORE] Close the inputStream in FileAppender when writin…

jhu-chang
authored and
fishcus
committed
[SPARK-36020][SQL][FOLLOWUP] RemoveRedundantProjects should retain the LOGICAL_PLAN_TAG tag

cloud-fan
authored and
fishcus
committed
[SPARK-28266][SQL] convertToLogicalRelation should not interpret `path` property when reading Hive tables

shardulm94
authored and
fishcus
committed
[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

yaooqinn
authored and
fishcus
committed
[SPARK-36242][CORE][3.1] Ensure spill file closed before set success = true in ExternalSorter.spillMemoryIteratorToDisk method

LuciferYang
authored and
fishcus
committed
[SPARK-36354][CORE] EventLogFileReader should skip rolling event log directories with no logs

dongjoon-hyun
authored and
fishcus
committed
[SPARK-36086][SQL][3.1] CollapseProject project replace alias should use origin column name

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36339][SQL][3.0] References to grouping that not part of aggregation should be replaced

gaoyajun02
authored and
fishcus
committed
[SPARK-36464][CORE] Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

kazuyukitanimura
authored and
fishcus
committed
[SPARK-36489][SQL] Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows

IonutBoicuAms
authored and
fishcus
committed
[SPARK-36353][SQL][3.1] RemoveNoopOperators should keep output schema

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36500][CORE] Fix temp_shuffle file leaking when a task is interrupted

jiangxb1987
authored and
fishcus
committed
[SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer

authored and
fishcus
committed
[SPARK-36564][CORE] Fix NullPointerException in LiveRDDDistribution.toApi

Ngone51
authored and
fishcus
committed
[SPARK-36352][SQL][3.1] Spark should check result plan's output schema name

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36509][CORE] Fix the issue that executors are never re-scheduled if the worker stops with standalone cluster

sarutak
authored and
fishcus
committed
[SPARK-36614][CORE][UI] Correct executor loss reason caused by decommission in UI

Ngone51
authored and
fishcus
committed
[SPARK-36639][SQL] Fix an issue that sequence builtin function causes ArrayIndexOutOfBoundsException if the arguments are under the condition of start == stop && step < 0

sarutak
authored and
fishcus
committed
[SPARK-36704][CORE] Expand exception handling to more Java 9 cases where reflection is limited at runtime, when reflecting to manage DirectByteBuffer settings

srowen
authored and
fishcus
committed
[SPARK-36715][SQL] InferFiltersFromGenerate should not infer filter for udf

cfmcgrady
authored and
fishcus
committed
[SPARK-36702][SQL] ArrayUnion handle duplicated Double.NaN and Float.Nan

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN and Float.NaN

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36755][SQL] ArraysOverlap should handle duplicated Double.NaN and Float.NaN

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36783][SQL] ScanOperation should not push Filter through nondeterministic Project

cloud-fan
authored and
fishcus
committed
[SPARK-36789][SQL] Use the correct constant type as the null value holder in array functions

cloud-fan
authored and
fishcus
committed
[SPARK-36741][SQL] ArrayDistinct handle duplicated Double.NaN and Float.Nan

AngersZhuuuu
authored and
fishcus
committed
[SPARK-35985][SQL][3.1] push partitionFilters for empty readDataSchema

huaxingao
authored and
fishcus
committed
[SPARK-36754][SQL] ArrayIntersect handle duplicated Double.NaN and Float.NaN

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36706][SQL][3.1] OverwriteByExpression conversion in DataSourceV2Strategy use wrong param in translateFilter

huaxingao
authored and
fishcus
committed
[SPARK-36803][SQL] Fix ArrayType conversion when reading Parquet files written in legacy mode

sadikovi
authored and
fishcus
committed
[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

authored and
fishcus
committed
[SPARK-36782][CORE][FOLLOW-UP] Only handle shuffle block in separate thread pool

Ngone51
authored and
fishcus
committed
[SPARK-36792][SQL] InSet should handle NaN

AngersZhuuuu
authored and
fishcus
committed
[SPARK-36874][SPARK-34634][SQL][3.1] ResolveReference.dedupRight should copy dataset_id tag to avoid ambiguous self join

sarutak
authored and
fishcus
committed
[SPARK-36717][CORE] Incorrect order of variable initialization may lead incorrect behavior

daugraph
authored and
fishcus
committed
[SPARK-23626][CORE] Eagerly compute RDD.partitions on entire DAG when submitting job to DAGScheduler

JoshRosen
authored and
fishcus
committed
[SPARK-37098][SQL][3.1] Alter table properties should invalidate cache

ulysses-you
authored and
fishcus
committed
[SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate

beliefer
authored and
fishcus
committed
[SPARK-37196][SQL] HiveDecimal enforcePrecisionScale failed return null

AngersZhuuuu
authored and
fishcus
committed
[SPARK-37388][SQL] Fix NPE in WidthBucket in WholeStageCodegenExec

tomvanbussel
authored and
fishcus
committed
[SPARK-37452][SQL][3.1] Char and Varchar break backward compatibility between v3.1 and v2

yaooqinn
authored and
fishcus
committed
[SPARK-37556][SQL] Deser void class fail with Java serialization

daijyc
authored and
fishcus
committed
[SPARK-37392][SQL] Fix the performance bug when inferring constraints for Generate

cloud-fan
authored and
fishcus
committed
[SPARK-37451][3.1][SQL] Fix cast string type to decimal type if spark.sql.legacy.allowNegativeScaleOfDecimal is enabled

wangyum
authored and
fishcus
committed
[SPARK-37060][CORE][3.1] Handle driver status response from backup masters

testsgmr
authored and
fishcus
committed
[SPARK-37654][SQL] Fix NPE in Row.getSeq when field is Null

huaxingao
authored and
fishcus
committed
[SPARK-37784][SQL] Correctly handle UDTs in CodeGenerator.addBufferedState()

JoshRosen
authored and
fishcus
committed
[SPARK-35714][FOLLOW-UP][CORE] WorkerWatcher should run System.exit in a thread out of RpcEnv

Ngone51
authored and
fishcus
committed
[SPARK-37860][UI] Fix taskindex in the stage page task event timeline

stczwd
authored and
fishcus
committed
[SPARK-34555][SQL][FOLLOWUP] code format
fishcus
committed

Commits on Feb 10, 2022

[SPARK-38075][SQL][3.1] Fix `hasNext` in `HiveScriptTransformationExec`'s process output iterator

bersprockets
authored and
fishcus
committed