Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
624 commits
Select commit Hold shift + click to select a range
ff14801
[SPARK-23510][SQL] Support Hive 2.2 and Hive 2.3 metastore
wangyum Mar 1, 2018
cdcccd7
[SPARK-23405] Generate additional constraints for Join's children
huleilei Mar 1, 2018
34811e0
[SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-core` dependenc…
dongjoon-hyun Mar 2, 2018
119f6a0
[SPARK-22883][ML][TEST] Streaming tests for spark.ml.feature, from A …
jkbradley Mar 2, 2018
0b6cead
[SPARKR][DOC] fix link in vignettes
felixcheung Mar 2, 2018
3a4d15e
[SPARK-23518][SQL] Avoid metastore access when the users only want to…
Mar 2, 2018
707e650
[SPARK-23097][SQL][SS] Migrate text socket source to V2
jerryshao Mar 2, 2018
487377e
[SPARK-23570][SQL] Add Spark 2.3.0 in HiveExternalCatalogVersionsSuite
gatorsmile Mar 2, 2018
9e26473
[SPARK-3159][ML] Add decision tree pruning
asolimando Mar 3, 2018
dea381d
[SPARK-23514][FOLLOW-UP] Remove more places using sparkContext.hadoop…
juliuszsompolski Mar 3, 2018
486f99e
[SPARK-23541][SS] Allow Kafka source to read data with greater parall…
tdas Mar 3, 2018
a89cdf5
[SQL][MINOR] XPathDouble prettyPrint should say 'double' not 'float'
ericl Mar 4, 2018
7965c91
[SPARK-23569][PYTHON] Allow pandas_udf to work with python3 style typ…
mstewart141 Mar 5, 2018
269cd53
[MINOR][DOCS] Fix a link in "Compatibility with Apache Hive"
HyukjinKwon Mar 5, 2018
2ce37b5
[SPARK-23546][SQL] Refactor stateless methods/values in CodegenContext
kiszk Mar 5, 2018
42cf48e
[SPARK-23496][CORE] Locality of coalesced partitions can be severely …
ala Mar 5, 2018
5ff72ff
[SPARK-23566][MINOR][DOC] Argument name mismatch fixed
animenon Mar 5, 2018
a366b95
[SPARK-23329][SQL] Fix documentation of trigonometric functions
misutoth Mar 5, 2018
947b4e6
[SPARK-23510][DOC][FOLLOW-UP] Update spark.sql.hive.metastore.version
wangyum Mar 5, 2018
4586ead
[SPARK-22430][R][DOCS] Unknown tag warnings when building R docs with…
rekhajoshm Mar 5, 2018
98a5c0a
[SPARK-22882][ML][TESTS] ML test for structured streaming: ml.classif…
WeichenXu123 Mar 5, 2018
ba622f4
[SPARK-23585][SQL] Add interpreted execution to UnwrapOption
mgaido91 Mar 5, 2018
b0f422c
[SPARK-23559][SS] Add epoch ID to DataWriterFactory.
jose-torres Mar 5, 2018
f2cab56
[SPARK-23040][CORE] Returns interruptible iterator for shuffle reader
advancedxy Mar 5, 2018
5085739
[SPARK-23538][CORE] Remove custom configuration for SSL client.
Mar 5, 2018
7706eea
[SPARK-18630][PYTHON][ML] Move del method from JavaParams to JavaWrap…
Mar 5, 2018
f6b49f9
[SPARK-23586][SQL] Add interpreted execution to WrapOption
mgaido91 Mar 6, 2018
8c5b34c
[SPARK-23604][SQL] Change Statistics.isEmpty to !Statistics.hasNonNul…
Mar 6, 2018
ad640a5
[SPARK-23303][SQL] improve the explain result for data source v2 rela…
cloud-fan Mar 6, 2018
e8a259d
[SPARK-23594][SQL] GetExternalRowField should support interpreted exe…
maropu Mar 6, 2018
8bceb89
[SPARK-23601][BUILD] Remove .md5 files from release
srowen Mar 6, 2018
4c587eb
[SPARK-23590][SQL] Add interpreted execution to CreateExternalRow
mgaido91 Mar 6, 2018
04e71c3
[MINOR][YARN] Add disable yarn.nodemanager.vmem-check-enabled option …
wangyum Mar 7, 2018
33c2cb2
[SPARK-23611][SQL] Add a helper function to check exception for expr …
maropu Mar 7, 2018
aff7d81
[SPARK-23591][SQL] Add interpreted execution to EncodeUsingSerializer
mgaido91 Mar 7, 2018
53561d2
[SPARK-23291][SQL][R] R's substr should not reduce starting position …
viirya Mar 7, 2018
c99fc9a
[SPARK-23550][CORE] Cleanup `Utils`.
Mar 7, 2018
ac76eff
[SPARK-23525][SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for exte…
jiangxb1987 Mar 7, 2018
77c91cc
[SPARK-23524] Big local shuffle blocks should not be checked for corr…
Mar 8, 2018
fe22f32
[SPARK-23620] Splitting thread dump lines by using the br tag
MaxGekk Mar 8, 2018
9bb239c
[SPARK-23159][PYTHON] Update cloudpickle to v0.4.3
BryanCutler Mar 8, 2018
d6632d1
[SPARK-23380][PYTHON] Adds a conf for Arrow fallback in toPandas/crea…
HyukjinKwon Mar 8, 2018
2cb23a8
[SPARK-23011][SQL][PYTHON] Support alternative function form with gro…
icexelloss Mar 8, 2018
7013eea
[SPARK-23522][PYTHON] always use sys.exit over builtin exit
benjaminp Mar 8, 2018
92e7ecb
[SPARK-23592][SQL] Add interpreted execution to DecodeUsingSerializer
mgaido91 Mar 8, 2018
3be4adf
[SPARK-22751][ML] Improve ML RandomForest shuffle performance
yangz27 Mar 8, 2018
ea48099
[SPARK-23628][SQL] calculateParamLength should not return 1 + num of …
mgaido91 Mar 8, 2018
e7bbca8
[SPARK-23602][SQL] PrintToStderr prints value also in interpreted mode
mgaido91 Mar 8, 2018
d90e77b
[SPARK-23271][SQL] Parquet output contains only _SUCCESS file after w…
dilipbiswal Mar 8, 2018
2c36736
[SPARK-23630][YARN] Allow user's hadoop conf customizations to take e…
Mar 9, 2018
2ca9bb0
[SPARK-23173][SQL] Avoid creating corrupt parquet files when loading …
mswit-databricks Mar 9, 2018
10b0657
[SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2
gengliangwang Mar 9, 2018
1a54f48
[SPARK-23510][SQL][FOLLOW-UP] Support Hive 2.2 and Hive 2.3 metastore
gatorsmile Mar 9, 2018
b6f837c
[PYTHON] Changes input variable to not conflict with built-in function
DylanGuedes Mar 10, 2018
b304e07
[SPARK-23462][SQL] improve missing field error message in `StructType`
xysun Mar 12, 2018
d5b41ae
[SPARK-23618][K8S][BUILD] Initialize BUILD_ARGS in docker-image-tool.sh
Mar 12, 2018
567bd31
[SPARK-23412][ML] Add cosine distance to BisectingKMeans
mgaido91 Mar 12, 2018
2337055
[SPARK-23656][TEST] Perform assertions in XXH64Suite.testKnownByteArr…
kiszk Mar 13, 2018
9ddd1e2
[MINOR][SQL][TEST] Create table using `dataSourceName` in `HadoopFsRe…
jiangxb1987 Mar 13, 2018
918fb9b
[SPARK-23547][SQL] Cleanup the .pipeout file when the Hive Session cl…
Mar 13, 2018
1098933
[SPARK-23598][SQL] Make methods in BufferedRowIterator public to avoi…
kiszk Mar 13, 2018
279b3db
[SPARK-22915][MLLIB] Streaming tests for spark.ml.feature, from N to Z
attilapiros Mar 15, 2018
4f5bad6
[SPARK-23642][DOCS] AccumulatorV2 subclass isZero scaladoc fix
Mar 15, 2018
7c3e899
[SPARK-23533][SS] Add support for changing ContinuousDataReader's sta…
xuanyuanking Mar 15, 2018
56e8f48
[SPARK-23695][PYTHON] Fix the error message for Kinesis streaming tests
HyukjinKwon Mar 15, 2018
15c3c98
[HOT-FIX] Fix SparkOutOfMemoryError: Unable to acquire 262144 bytes o…
wangyum Mar 15, 2018
7618896
[SPARK-23658][LAUNCHER] InProcessAppHandle uses the wrong class in ge…
Mar 16, 2018
18f8575
[SPARK-23671][CORE] Fix condition to enable the SHS thread pool.
Mar 16, 2018
3675af7
[SPARK-23608][CORE][WEBUI] Add synchronization in SHS between attachS…
zhouyejoe Mar 16, 2018
c2632ed
[SPARK-23670][SQL] Fix memory leak on SparkPlanGraphWrapper
myroslavlisniak Mar 16, 2018
ca83526
[SPARK-23644][CORE][UI] Use absolute path for REST call in SHS
mgaido91 Mar 16, 2018
c952000
[SPARK-23635][YARN] AM env variable should not overwrite same name en…
jerryshao Mar 16, 2018
5414abc
[SPARK-23553][TESTS] Tests should not assume the default value of `sp…
dongjoon-hyun Mar 16, 2018
dffeac3
[SPARK-18371][STREAMING] Spark Streaming backpressure generates batch…
Mar 16, 2018
88d8de9
[SPARK-23581][SQL] Add interpreted unsafe projection
hvanhovell Mar 16, 2018
9945b02
[SPARK-23680] Fix entrypoint.sh to properly support Arbitrary UIDs
Mar 16, 2018
bd201bf
[SPARK-23623][SS] Avoid concurrent use of cached consumers in CachedK…
tdas Mar 16, 2018
8a72734
[SPARK-15009][PYTHON][ML] Construct a CountVectorizerModel from a voc…
BryanCutler Mar 16, 2018
8a1efe3
[SPARK-23683][SQL] FileCommitProtocol.instantiate() hardening
steveloughran Mar 16, 2018
61487b3
[SPARK-23706][PYTHON] spark.conf.get(value, default=None) should prod…
HyukjinKwon Mar 18, 2018
745c8c0
[SPARK-23708][CORE] Correct comment for function addShutDownHook in S…
caneGuy Mar 19, 2018
4de638c
[SPARK-23599][SQL] Add a UUID generator from Pseudo-Random Numbers
viirya Mar 19, 2018
f15906d
[SPARK-22839][K8S] Remove the use of init-container for downloading r…
ifilonenko Mar 19, 2018
5f4deff
[SPARK-23660] Fix exception in yarn cluster mode when application end…
gaborgsomogyi Mar 20, 2018
5663218
[SPARK-23691][PYTHON] Use sql_conf util in PySpark tests where possible
HyukjinKwon Mar 20, 2018
5e7bc2a
[SPARK-23649][SQL] Skipping chars disallowed in UTF-8
MaxGekk Mar 20, 2018
7f5e8aa
[SPARK-21898][ML] Feature parity for KolmogorovSmirnovTest in MLlib
WeichenXu123 Mar 20, 2018
2c4b996
[SPARK-23574][SQL] Report SinglePartition in DataSourceV2ScanExec whe…
jose-torres Mar 20, 2018
477d6bd
[SPARK-23500][SQL] Fix complex type simplification rules to apply to …
henryr Mar 20, 2018
983e8d9
[SPARK-23666][SQL] Do not display exprIds of Alias in user-facing info.
maropu Mar 21, 2018
500b21c
[SPARK-23568][ML] Use metadata numAttributes if available in Silhouette
mgaido91 Mar 21, 2018
bf09f2f
[SPARK-10884][ML] Support prediction on single instance for regressio…
WeichenXu123 Mar 21, 2018
8d79113
[SPARK-23577][SQL] Supports custom line separator for text datasource
HyukjinKwon Mar 21, 2018
98d0ea3
[SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out
maropu Mar 21, 2018
918c7e9
[SPARK-23288][SS] Fix output metrics with parquet sink
gaborgsomogyi Mar 21, 2018
2b89e4a
[SPARK-18580][DSTREAM][KAFKA] Add spark.streaming.backpressure.initia…
Mar 21, 2018
a091ee6
[MINOR] Fix Java lint from new JavaKolmogorovSmirnovTestSuite
jkbradley Mar 21, 2018
0604bea
[SPARK-23729][CORE] Respect URI fragment when resolving globs
misutoth Mar 22, 2018
95e51ff
[SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should …
rednaxelafx Mar 22, 2018
5c9eaa6
[SPARK-23372][SQL] Writing empty struct in parquet fails during execu…
dilipbiswal Mar 22, 2018
4d37008
[SPARK-23599][SQL] Use RandomUUIDGenerator in Uuid expression
viirya Mar 22, 2018
a649fcf
[MINOR][PYTHON] Remove unused codes in schema parsing logics of PySpark
HyukjinKwon Mar 23, 2018
b2edc30
[SPARK-23614][SQL] Fix incorrect reuse exchange when caching is used
viirya Mar 23, 2018
5fa4384
[SPARK-23361][YARN] Allow AM to restart after initial tokens expire.
Mar 23, 2018
92e9525
[MINOR][R] Fix R lint failure
HyukjinKwon Mar 23, 2018
6ac4fba
[SPARK-23769][CORE] Remove comments that unnecessarily disable Scalas…
arucard21 Mar 23, 2018
8b56f16
[SPARK-23759][UI] Unable to bind Spark UI to specific host name / IP
Mar 23, 2018
cb43bbe
[SPARK-21685][PYTHON][ML] PySpark Params isSet state should not chang…
BryanCutler Mar 23, 2018
95c03cb
[SPARK-23783][SPARK-11239][ML] Add PMML export to Spark ML pipelines
holdenk Mar 23, 2018
a336553
[SPARK-23615][ML][PYSPARK] Add maxDF Parameter to Python CountVectorizer
huaxingao Mar 23, 2018
816a549
[SPARK-23788][SS] Fix race in StreamingQuerySuite
jose-torres Mar 25, 2018
5f653d4
[SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQuerySuite
maropu Mar 25, 2018
e4bec7c
[SPARK-23549][SQL] Cast to timestamp when comparing timestamp with date
kiszk Mar 25, 2018
a9350d7
[SPARK-23700][PYTHON] Cleanup imports in pyspark.sql
BryanCutler Mar 26, 2018
087fb31
[SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE `pandas_udf` with keyw…
mstewart141 Mar 26, 2018
eb48edf
[SPARK-23787][TESTS] Fix file download test in SparkSubmitSuite for H…
Mar 26, 2018
b30a7d2
[SPARK-23572][DOCS] Bring "security.md" up to date.
Mar 26, 2018
3e778f5
[SPARK-23162][PYSPARK][ML] Add r2adj into Python API in LinearRegress…
kevinyu98 Mar 26, 2018
35997b5
[SPARK-23794][SQL] Make UUID as stateful expression
viirya Mar 27, 2018
c68ec4e
[SPARK-23096][SS] Migrate rate source to V2
jerryshao Mar 27, 2018
ed72bad
[SPARK-23699][PYTHON][SQL] Raise same type of error caught with Arrow…
BryanCutler Mar 28, 2018
34c4b9c
[SPARK-23765][SQL] Supports custom line separator for json datasource
HyukjinKwon Mar 28, 2018
761565a
Revert "[SPARK-23096][SS] Migrate rate source to V2"
gatorsmile Mar 28, 2018
ea2fdc0
[SPARK-23675][WEB-UI] Title add spark logo, use spark logo image
Mar 29, 2018
641aec6
[SPARK-23806] Broadcast.unpersist can cause fatal exception when used…
Mar 29, 2018
505480c
[SPARK-23770][R] Exposes repartitionByRange in SparkR
HyukjinKwon Mar 29, 2018
491ec11
[SPARK-23785][LAUNCHER] LauncherBackend doesn't check state of connec…
Mar 29, 2018
a7755fd
[SPARK-23639][SQL] Obtain token before init metastore client in Spark…
yaooqinn Mar 29, 2018
b348901
[SPARK-23808][SQL] Set default Spark session in test-only spark sessi…
jose-torres Mar 30, 2018
df05fb6
[SPARK-23743][SQL] Changed a comparison logic from containing 'slf4j'…
jongyoul Mar 30, 2018
b02e76c
[SPARK-23727][SQL] Support for pushing down filters for DateType in p…
yucai Mar 30, 2018
5b5a36e
Roll forward "[SPARK-23096][SS] Migrate rate source to V2"
jose-torres Mar 30, 2018
bc8d093
[SPARK-23500][SQL][FOLLOWUP] Fix complex type simplification rules to…
gatorsmile Mar 30, 2018
ae91720
[SPARK-23640][CORE] Fix hadoop config may override spark config
wangyum Mar 30, 2018
15298b9
[SPARK-23827][SS] StreamingJoinExec should ensure that input data is …
tdas Mar 30, 2018
529f847
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
jiangxb1987 Mar 31, 2018
44a9f8e
[SPARK-15009][PYTHON][FOLLOWUP] Add default param checks for CountVec…
BryanCutler Apr 2, 2018
6151f29
[SPARK-23825][K8S] Requesting memory + memory overhead for pod memory
dvogelbacher Apr 2, 2018
fe2b7a4
[SPARK-23285][K8S] Add a config property for specifying physical exec…
liyinan926 Apr 2, 2018
a7c19d9
[SPARK-23713][SQL] Cleanup UnsafeWriter and BufferHolder classes
kiszk Apr 2, 2018
28ea4e3
[SPARK-23834][TEST] Wait for connection before disconnect in Launcher…
Apr 2, 2018
a135182
[SPARK-23690][ML] Add handleinvalid to VectorAssembler
Apr 2, 2018
441d0d0
[SPARK-19964][CORE] Avoid reading from remote repos in SparkSubmitSuite.
Apr 3, 2018
8020f66
[MINOR][DOC] Fix a few markdown typos
Apr 3, 2018
7cf9fab
[MINOR][CORE] Show block manager id when remove RDD/Broadcast fails.
jiangxb1987 Apr 3, 2018
66a3a5a
[SPARK-23099][SS] Migrate foreach sink to DataSourceV2
jose-torres Apr 3, 2018
1035aaa
[SPARK-23587][SQL] Add interpreted execution for MapObjects expression
viirya Apr 3, 2018
359375e
[SPARK-23809][SQL] Active SparkSession should be set by getOrCreate
ericl Apr 4, 2018
5cfd5fa
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unr…
Apr 4, 2018
16ef6ba
[SPARK-23826][TEST] TestHiveSparkSession should set default session
gatorsmile Apr 4, 2018
5197562
[SPARK-21351][SQL] Update nullability based on children's output
maropu Apr 4, 2018
a355236
[SPARK-23583][SQL] Invoke should support interpreted execution
kiszk Apr 4, 2018
cccaaa1
[SPARK-23668][K8S] Add config option for passing through k8s Pod.spec…
Apr 4, 2018
d8379e5
[SPARK-23838][WEBUI] Running SQL query is displayed as "completed" in…
gengliangwang Apr 4, 2018
d3bd043
[SPARK-23637][YARN] Yarn might allocate more resource if a same execu…
Apr 4, 2018
c5c8b54
[SPARK-23593][SQL] Add interpreted execution for InitializeJavaBean e…
viirya Apr 5, 2018
1822ecd
[SPARK-23582][SQL] StaticInvoke should support interpreted execution
kiszk Apr 5, 2018
b2329fb
Revert "[SPARK-23593][SQL] Add interpreted execution for InitializeJa…
hvanhovell Apr 5, 2018
d9ca1c9
[SPARK-23593][SQL] Add interpreted execution for InitializeJavaBean e…
viirya Apr 5, 2018
4807d38
[SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks to choose se…
kiszk Apr 6, 2018
f2ac087
[SPARK-23870][ML] Forward RFormula handleInvalid Param to VectorAssem…
Apr 6, 2018
d65e531
[SPARK-23823][SQL] Keep origin in transformExpression
Apr 6, 2018
249007e
[SPARK-19724][SQL] create a managed table with an existed default tab…
gengliangwang Apr 6, 2018
6ade5cb
[MINOR][DOC] Fix some typos and grammar issues
dsakuma Apr 6, 2018
9452401
[SPARK-23822][SQL] Improve error message for Parquet schema mismatches
yuchenhuo Apr 6, 2018
d766ea2
[SPARK-23861][SQL][DOC] Clarify default window frame with and without…
icexelloss Apr 6, 2018
c926acf
[SPARK-23882][CORE] UTF8StringSuite.writeToOutputStreamUnderflow() is…
kiszk Apr 6, 2018
d23a805
[SPARK-23859][ML] Initial PR for Instrumentation improvements: UUID a…
MrBago Apr 6, 2018
b6935ff
[SPARK-10399][SPARK-23879][HOTFIX] Fix Java lint errors
kiszk Apr 6, 2018
e998250
[SPARK-23828][ML][PYTHON] PySpark StringIndexerModel should have cons…
huaxingao Apr 6, 2018
6ab134c
[SPARK-21898][ML][FOLLOWUP] Fix Scala 2.12 build.
ueshin Apr 6, 2018
2c1fe64
[SPARK-23847][PYTHON][SQL] Add asc_nulls_first, asc_nulls_last to PyS…
huaxingao Apr 8, 2018
6a73457
[SPARK-23849][SQL] Tests for the samplingRatio option of JSON datasource
MaxGekk Apr 8, 2018
710a68c
[SPARK-23892][TEST] Improve converge and fix lint error in UTF8String…
kiszk Apr 8, 2018
8d40a79
[SPARK-23893][CORE][SQL] Avoid possible integer overflow in multiplic…
kiszk Apr 8, 2018
32471ba
Fix typo in Python docstring kinesis example
Apr 9, 2018
d81f29e
[SPARK-23881][CORE][TEST] Fix flaky test JobCancellationSuite."interr…
jiangxb1987 Apr 9, 2018
10f45bb
[SPARK-23816][CORE] Killed tasks should ignore FetchFailures.
squito Apr 9, 2018
7c1654e
[SPARK-22856][SQL] Add wrappers for codegen output and nullability
viirya Apr 9, 2018
252468a
[SPARK-14681][ML] Provide label/impurity stats for spark.ml decision …
WeichenXu123 Apr 9, 2018
61b7247
[INFRA] Close stale PRs.
Apr 9, 2018
f94f362
[SPARK-23947][SQL] Add hashUTF8String convenience method to hasher cl…
rednaxelafx Apr 10, 2018
6498884
[SPARK-23898][SQL] Simplify add & subtract code generation
hvanhovell Apr 10, 2018
95034af
[SPARK-23841][ML] NodeIdCache should unpersist the last cached nodeId…
zhengruifeng Apr 10, 2018
3323b15
[SPARK-23864][SQL] Add unsafe object writing to UnsafeWriter
hvanhovell Apr 10, 2018
e179658
[SPARK-19724][SQL][FOLLOW-UP] Check location of managed table when ig…
gengliangwang Apr 10, 2018
adb222b
[SPARK-23751][ML][PYSPARK] Kolmogorov-Smirnoff test Python API in pys…
WeichenXu123 Apr 10, 2018
4f1e8b9
[SPARK-23871][ML][PYTHON] add python api for VectorAssembler handleIn…
huaxingao Apr 10, 2018
7c7570d
[SPARK-23944][ML] Add the set method for the two LSHModel
lu-wang-dl Apr 11, 2018
c7622be
[SPARK-23847][FOLLOWUP][PYTHON][SQL] Actually test [desc|acs]_nulls_[…
HyukjinKwon Apr 11, 2018
87611bb
[MINOR][DOCS] Fix R documentation generation instruction for roxygen2
HyukjinKwon Apr 11, 2018
c604d65
[SPARK-23951][SQL] Use actual java class instead of string representa…
hvanhovell Apr 11, 2018
271c891
[SPARK-23960][SQL][MINOR] Mark HashAggregateExec.bufVars as transient
rednaxelafx Apr 11, 2018
653fe02
[SPARK-6951][CORE] Speed up parsing of event logs during listing.
Apr 11, 2018
3cb8204
[SPARK-22941][CORE] Do not exit JVM when submit fails with in-process…
Apr 11, 2018
75a1830
[SPARK-22883] ML test for StructuredStreaming: spark.ml.feature, I-M
jkbradley Apr 11, 2018
9d960de
typo rawPredicition changed to rawPrediction
JBauerKogentix Apr 11, 2018
e904dfa
Revert "[SPARK-23960][SQL][MINOR] Mark HashAggregateExec.bufVars as t…
gatorsmile Apr 12, 2018
6a2289e
[SPARK-23962][SQL][TEST] Fix race in currentExecutionIds().
squito Apr 12, 2018
0b19122
[SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock
kiszk Apr 12, 2018
0f93b91
[SPARK-23751][FOLLOW-UP] fix build for scala-2.12
WeichenXu123 Apr 12, 2018
682002b
[SPARK-23867][SCHEDULER] use droppedCount in logWarning
Apr 13, 2018
14291b0
[SPARK-23748][SS] Fix SS continuous process doesn't support SubqueryA…
jerryshao Apr 13, 2018
ab7b961
[SPARK-23942][PYTHON][SQL] Makes collect in PySpark as action for a q…
HyukjinKwon Apr 13, 2018
1018be4
[SPARK-23971] Should not leak Spark sessions across test suites
ericl Apr 13, 2018
4b07036
[SPARK-23815][CORE] Spark writer dynamic partition overwrite mode may…
Apr 13, 2018
0323e61
[SPARK-23905][SQL] Add UDF weekday
yucai Apr 13, 2018
a83ae0d
[SPARK-22839][K8S] Refactor to unify driver and executor pod builder …
mccheah Apr 13, 2018
4dfd746
[SPARK-23896][SQL] Improve PartitioningAwareFileIndex
gengliangwang Apr 13, 2018
25892f3
[SPARK-23375][SQL] Eliminate unneeded Sort in Optimizer
mgaido91 Apr 13, 2018
558f31b
[SPARK-23963][SQL] Properly handle large number of columns in query o…
bersprockets Apr 13, 2018
cbb41a0
[SPARK-23966][SS] Refactoring all checkpoint file writing logic in a …
tdas Apr 13, 2018
73f2853
[SPARK-23979][SQL] MultiAlias should not be a CodegenFallback
viirya Apr 14, 2018
c096493
[SPARK-23956][YARN] Use effective RPC port in AM registration
gerashegalov Apr 16, 2018
6931022
[SPARK-23917][SQL] Add array_max function
mgaido91 Apr 16, 2018
083cf22
[SPARK-21033][CORE][FOLLOW-UP] Update Spillable
wangyum Apr 16, 2018
5003736
[SPARK-9312][ML] Add RawPrediction, numClasses, and numFeatures for O…
lu-wang-dl Apr 16, 2018
0461482
[SPARK-21088][ML] CrossValidator, TrainValidationSplit support collec…
WeichenXu123 Apr 16, 2018
fd990a9
[SPARK-23873][SQL] Use accessors in interpreted LambdaVariable
viirya Apr 16, 2018
14844a6
[SPARK-23918][SQL] Add array_min function
mgaido91 Apr 17, 2018
1cc66a0
[SPARK-23687][SS] Add a memory source for continuous processing.
jose-torres Apr 17, 2018
05ae747
[SPARK-23747][STRUCTURED STREAMING] Add EpochCoordinator unit tests
Apr 17, 2018
30ffb53
[SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
viirya Apr 17, 2018
0a9172a
[SPARK-23835][SQL] Add not-null check to Tuples' arguments deserializ…
mgaido91 Apr 17, 2018
ed4101d
[SPARK-22676] Avoid iterating all partition paths when spark.sql.hive…
Apr 17, 2018
3990daa
[SPARK-23948] Trigger mapstage's job listener in submitMissingTasks
Apr 17, 2018
f39e82c
[SPARK-23986][SQL] freshName can generate non-unique names
mgaido91 Apr 17, 2018
1ca3c50
[SPARK-21741][ML][PYSPARK] Python API for DataFrame-based multivariat…
WeichenXu123 Apr 17, 2018
5fccdae
[SPARK-22968][DSTREAM] Throw an exception on partition revoking issue
jerryshao Apr 18, 2018
1e3b876
[SPARK-21479][SQL] Outer join filter pushdown in null supplying table…
maryannxue Apr 18, 2018
310a8cd
[SPARK-23341][SQL] define some standard options for data source v2
cloud-fan Apr 18, 2018
cce4694
[SPARK-24002][SQL] Task not serializable caused by org.apache.parquet…
gatorsmile Apr 18, 2018
f81fa47
[SPARK-23926][SQL] Extending reverse function to support ArrayType ar…
Apr 18, 2018
f09a9e9
[SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might g…
ueshin Apr 18, 2018
a906647
[SPARK-23875][SQL][FOLLOWUP] Add IndexedSeq wrapper for ArrayData
viirya Apr 18, 2018
0c94e48
[SPARK-23775][TEST] Make DataFrameRangeSuite not flaky
gaborgsomogyi Apr 18, 2018
8bb0df2
[SPARK-24014][PYSPARK] Add onStreamingStarted method to StreamingList…
viirya Apr 19, 2018
d5bec48
[SPARK-23919][SQL] Add array_position function
kiszk Apr 19, 2018
46bb2b5
[SPARK-23924][SQL] Add element_at function
kiszk Apr 19, 2018
1b08c43
[SPARK-23584][SQL] NewInstance should support interpreted execution
maropu Apr 19, 2018
e134165
[SPARK-23588][SQL] CatalystToExternalMap should support interpreted e…
maropu Apr 19, 2018
9e10f69
[SPARK-22676][FOLLOW-UP] fix code style for test.
Apr 19, 2018
d96c3e3
[SPARK-21811][SQL] Fix the inconsistency behavior when finding the wi…
jiangxb1987 Apr 19, 2018
0deaa52
[SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForF…
Ngone51 Apr 19, 2018
6e19f76
[SPARK-23989][SQL] exchange should copy data before non-serialized sh…
cloud-fan Apr 19, 2018
a471880
[SPARK-24026][ML] Add Power Iteration Clustering to spark.ml
wangmiao1981 Apr 19, 2018
9ea8d3d
[SPARK-22362][SQL] Add unit test for Window Aggregate Functions
attilapiros Apr 19, 2018
e55953b
[SPARK-24022][TEST] Make SparkContextSuite not flaky
gaborgsomogyi Apr 19, 2018
b3fde5a
[SPARK-23877][SQL] Use filter predicates to prune partitions in metad…
rdblue Apr 20, 2018
e6b4660
[SPARK-23736][SQL] Extending the concat function to support array col…
Apr 20, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-23594][SQL] GetExternalRowField should support interpreted exe…
…cution

## What changes were proposed in this pull request?
This pr added interpreted execution for `GetExternalRowField`.

## How was this patch tested?
Added tests in `ObjectExpressionsSuite`.

Author: Takeshi Yamamuro <[email protected]>

Closes apache#20746 from maropu/SPARK-23594.
  • Loading branch information
maropu authored and hvanhovell committed Mar 6, 2018
commit e8a259d66dda0d4c76f3af8933676bade8a7451d
Original file line number Diff line number Diff line change
Expand Up @@ -1358,11 +1358,19 @@ case class GetExternalRowField(

override def dataType: DataType = ObjectType(classOf[Object])

override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")

private val errMsg = s"The ${index}th field '$fieldName' of input row cannot be null."

override def eval(input: InternalRow): Any = {
val inputRow = child.eval(input).asInstanceOf[Row]
if (inputRow == null) {
throw new RuntimeException("The input external row cannot be null.")
}
if (inputRow.isNullAt(index)) {
throw new RuntimeException(errMsg)
}
inputRow.get(index)
}

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
// Use unnamed reference that doesn't create a local field here to reduce the number of fields
// because errMsgField is used only when the field is null.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.sql.catalyst.expressions

import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.Row
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.catalyst.expressions.objects._
Expand Down Expand Up @@ -84,4 +85,23 @@ class ObjectExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkEvaluation(wrapObject, expected, InternalRow.fromSeq(Seq(input)))
}
}

test("SPARK-23594 GetExternalRowField should support interpreted execution") {
val inputObject = BoundReference(0, ObjectType(classOf[Row]), nullable = true)
val getRowField = GetExternalRowField(inputObject, index = 0, fieldName = "c0")
Seq((Row(1), 1), (Row(3), 3)).foreach { case (input, expected) =>
checkEvaluation(getRowField, expected, InternalRow.fromSeq(Seq(input)))
}

// If an input row or a field are null, a runtime exception will be thrown
val errMsg1 = intercept[RuntimeException] {
evaluate(getRowField, InternalRow.fromSeq(Seq(null)))
}.getMessage
assert(errMsg1 === "The input external row cannot be null.")

val errMsg2 = intercept[RuntimeException] {
evaluate(getRowField, InternalRow.fromSeq(Seq(Row(null))))
}.getMessage
assert(errMsg2 === "The 0th field 'c0' of input row cannot be null.")
}
}