Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
846 commits
Select commit Hold shift + click to select a range
8141d55
[SPARK-23633][SQL] Update Pandas UDFs section in sql-programming-guide
icexelloss Jul 31, 2018
b4fd75f
[SPARK-24972][SQL] PivotFirst could not handle pivot columns of compl…
maryannxue Jul 31, 2018
4ac2126
[SPARK-24536] Validate that an evaluated limit clause cannot be null
mauropalsgraaf Jul 31, 2018
1223a20
[SPARK-24609][ML][DOC] PySpark/SparkR doc doesn't explain RandomFores…
zhengruifeng Jul 31, 2018
e82784d
[SPARK-18057][SS] Update Kafka client version from 0.10.0.1 to 2.0.0
tedyu Jul 31, 2018
42dfe4f
[SPARK-24973][PYTHON] Add numIter to Python ClusteringSummary
huaxingao Jul 31, 2018
f4772fd
[SPARK-24976][PYTHON] Allow None for Decimal type conversion (specifi…
HyukjinKwon Aug 1, 2018
5f3441e
[SPARK-24893][SQL] Remove the entire CaseWhen if all the outputs are …
dbtsai Aug 1, 2018
1f7e22c
[SPARK-24951][SQL] Table valued functions should throw AnalysisException
rxin Aug 1, 2018
1efffb7
[SPARK-24982][SQL] UDAF resolution should not throw AssertionError
rxin Aug 1, 2018
1122754
[SPARK-24653][TESTS] Avoid cross-job pollution in TestUtils / SpillLi…
Aug 1, 2018
defc54c
[SPARK-24971][SQL] remove SupportsDeprecatedScanRow
cloud-fan Aug 1, 2018
95a9d5e
[SPARK-23915][SQL] Add array_except function
kiszk Aug 1, 2018
f5113ea
[SPARK-24960][K8S] explicitly expose ports on driver container
Aug 1, 2018
9f55860
[SPARK-24937][SQL] Datasource partition table should load empty stati…
wangyum Aug 1, 2018
ce084d3
[SPARK-24990][SQL] merge ReadSupport and ReadSupportWithSchema
cloud-fan Aug 1, 2018
c5fe412
[SPARK-18188][DOC][FOLLOW-UP] Add `spark.broadcast.checksum` to confi…
10110346 Aug 2, 2018
c9914cf
[MINOR][DOCS] Add note about Spark network security
srowen Aug 2, 2018
166f346
[SPARK-24957][SQL][FOLLOW-UP] Clean the code for AVERAGE
gatorsmile Aug 2, 2018
57d9949
[SPARK-24557][ML] ClusteringEvaluator support array input
zhengruifeng Aug 2, 2018
2754157
[SPARK-24795][CORE][FOLLOWUP] Kill all running tasks when a task in a…
jiangxb1987 Aug 2, 2018
a657369
[SPARK-14540][CORE] Fix remaining major issues for Scala 2.12 Support
Aug 2, 2018
46110a5
[SPARK-24865][FOLLOW-UP] Remove AnalysisBarrier LogicalPlan Node
gatorsmile Aug 2, 2018
7be6fc3
[SPARK-24742] Fix NullPointerexception in Field Metadata
kupferk Aug 2, 2018
d182b3d
[SPARK-24742] Fix NullPointerexception in Field Metadata
kupferk Aug 2, 2018
f04cd67
[MINOR] remove dead code in ExpressionEvalHelper
cloud-fan Aug 2, 2018
15fc237
Updates to Accumulators
LucaCanali Jul 18, 2018
ad2e636
[SPARK-24598][DOCS] State in the documentation the behavior when arit…
mgaido91 Aug 2, 2018
38e4699
[SPARK-24820][SPARK-24821][CORE] Fail fast when submitted job contain…
jiangxb1987 Aug 2, 2018
0df6bf8
[BUILD] Fix lint-python.
ueshin Aug 2, 2018
02f9677
[SPARK-23908][SQL] Add transform function.
ueshin Aug 2, 2018
efef553
[SPARK-24705][SQL] ExchangeCoordinator broken when duplicate exchange…
maropu Aug 2, 2018
d0bc3ed
[SPARK-24896][SQL] Uuid should produce different values for each exec…
viirya Aug 2, 2018
bbdcc3b
[SPARK-22219][SQL] Refactor code to get a value for "spark.sql.codege…
kiszk Aug 2, 2018
29077a1
[SPARK-24795][CORE][FOLLOWUP] Combine BarrierTaskContext with Barrier…
jiangxb1987 Aug 3, 2018
7cf16a7
[SPARK-24773] Avro: support logical timestamp type with different pre…
gengliangwang Aug 3, 2018
b3f2911
[SPARK-24945][SQL] Switching to uniVocity 2.7.3
MaxGekk Aug 3, 2018
73dd6cf
[SPARK-24966][SQL] Implement precedence rules for set operations.
dilipbiswal Aug 3, 2018
f45d60a
[SPARK-25002][SQL] Avro: revise the output record namespace
gengliangwang Aug 3, 2018
b0d6967
[SPARK-24788][SQL] RelationalGroupedDataset.toString with unresolved …
c-horn Aug 3, 2018
19a4531
[SPARK-24997][SQL] Enable support of MINUS ALL
dilipbiswal Aug 3, 2018
ebf33a3
[SAPRK-25011][ML] add prefix to __all__ in fpm.py
YY-OnCall Aug 3, 2018
53ca975
[SPARK-25009][CORE] Standalone Cluster mode application submit is not…
Aug 3, 2018
273b284
[SPARK-24993][SQL] Make Avro Fast Again
dbtsai Aug 3, 2018
c32dbd6
[SPARK-18057][FOLLOW-UP][SS] Update Kafka client version from 0.10.0.…
srowen Aug 3, 2018
92b4884
[SPARK-24954][CORE] Fail fast on job submit if run a barrier stage wi…
jiangxb1987 Aug 3, 2018
8c14276
Little typo
dongido001 Aug 3, 2018
4c27663
[SPARK-18057][FOLLOW-UP][SS] Update Kafka client version from 0.10.0.…
srowen Aug 3, 2018
41c2227
[SPARK-24722][SQL] pivot() with Column type argument
MaxGekk Aug 4, 2018
36ea55e
[SPARK-24940][SQL] Coalesce and Repartition Hint for SQL Queries
jzhuge Aug 4, 2018
0ecc132
[SPARK-23909][SQL] Add filter function.
ueshin Aug 4, 2018
684c719
[SPARK-23915][SQL][FOLLOWUP] Add array_except function
cloud-fan Aug 4, 2018
70462f2
[SPARK-24926][CORE] Ensure numCores is used consistently in all netty…
NiharS Aug 4, 2018
55e3ae6
[SPARK-25001][BUILD] Fix miscellaneous build warnings
HyukjinKwon Aug 4, 2018
b7fdf8e
[SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for T…
Aug 4, 2018
5f9633d
[SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
srowen Aug 4, 2018
327bb30
[SPARK-23911][SQL] Add aggregate function.
ueshin Aug 4, 2018
ac527b5
[SPARK-24991][SQL] use InternalRow in DataSourceWriter
cloud-fan Aug 6, 2018
64ad7b8
[SPARK-23772][FOLLOW-UP][SQL] Provide an option to ignore column of a…
MaxGekk Aug 6, 2018
d063e3a
[SPARK-24940][SQL] Use IntegerLiteral in ResolveCoalesceHints
jzhuge Aug 6, 2018
c1760da
[SPARK-25025][SQL] Remove the default value of isAll in INTERSECT/EXCEPT
dilipbiswal Aug 6, 2018
35700bb
[SPARK-24981][CORE] ShutdownHook timeout causes job to fail when succ…
Aug 6, 2018
1a5e460
[SPARK-23913][SQL] Add array_intersect function
kiszk Aug 6, 2018
51e2b38
[SPARK-24992][CORE] spark should randomize yarn local dir selection
Aug 6, 2018
278984d
[SPARK-25019][BUILD] Fix orc dependency to use the same exclusion rules
dongjoon-hyun Aug 6, 2018
3c96937
[SPARK-24948][SHS] Delegate check access permissions to the file system
mgaido91 Aug 6, 2018
87ca739
[SPARK-24161][SS] Enable debug package feature on structured streaming
HeartSaVioR Aug 6, 2018
408a3ff
[SPARK-25036][SQL] Should compare ExprValue.isNull with LiteralTrue/L…
kiszk Aug 6, 2018
0f3fa2f
[SPARK-24996][SQL] Use DSL in DeclarativeAggregate
mgaido91 Aug 6, 2018
1076e4f
[MINOR][DOCS] Fix grammatical error in SortShuffleManager
deshanxiao Aug 7, 2018
6afe6f3
[SPARK-24637][SS] Add metrics regarding state and watermark to dropwi…
HeartSaVioR Aug 7, 2018
18b6ec1
[SPARK-24748][SS] Support for reporting custom metrics via StreamingQ…
arunmahadevan Aug 7, 2018
51bee7a
[SPARK-25018][INFRA] Use `Co-authored-by` and `Signed-off-by` git tra…
dbtsai Aug 7, 2018
4446a0b
[SPARK-23914][SQL][FOLLOW-UP] refactor ArrayUnion
kiszk Aug 7, 2018
4376362
[SPARK-25010][SQL] Rand/Randn should produce different values for eac…
viirya Aug 7, 2018
388f5a0
[SPARK-24817][CORE] Implement BarrierTaskContext.barrier()
jiangxb1987 Aug 7, 2018
88e0c7b
[SPARK-24341][SQL] Support only IN subqueries with the same number of…
mgaido91 Aug 7, 2018
131ca14
[SPARK-24005][CORE] Remove usage of Scala’s parallel collection
MaxGekk Aug 7, 2018
819c4de
[SPARK-24772][SQL] Avro: support logical date type
gengliangwang Aug 7, 2018
b4bf8be
[SPARK-19602][SQL] Support column resolution of fully qualified colum…
skambha Aug 7, 2018
6a143e3
[SPARK-23928][TESTS][FOLLOWUP] Set seed to avoid flakiness
mgaido91 Aug 7, 2018
1a29fec
[SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp
cloud-fan Aug 7, 2018
298e80f
[SPARK-25041][BUILD] upgrade genJavaDoc-plugin from 0.10 to 0.11
kiszk Aug 7, 2018
cb6cb31
[SPARK-23937][SQL] Add map_filter SQL function
mgaido91 Aug 7, 2018
8c13cb2
[SPARK-25031][SQL] Fix MapType schema print
invkrh Aug 7, 2018
f6356f9
[SPARK-25046][SQL] Fix Alter View can excute sql like "ALTER VIEW ...…
sddyljsx Aug 7, 2018
66699c5
[SPARK-25029][TESTS] Scala 2.12 issues: TaskNotSerializable and Janin…
srowen Aug 7, 2018
d90f133
[SPARK-25045][CORE] Make `RDDBarrier.mapParititions` similar to `RDD.…
jiangxb1987 Aug 8, 2018
5fef6e3
[SPARK-24251][SQL] Add AppendData logical plan.
rdblue Aug 8, 2018
c7a229d
[SPARK-25010][SQL][FOLLOWUP] Shuffle should also produce different va…
ueshin Aug 8, 2018
f08f6f4
[SPARK-23935][SQL][FOLLOWUP] mapEntry throws org.codehaus.commons.com…
kiszk Aug 8, 2018
960af63
[SPARK-25036][SQL] avoid match may not be exhaustive in Scala-2.12
kiszk Aug 8, 2018
6f6a420
[SPARK-23911][SQL][FOLLOW-UP] Fix examples of aggregate function.
ueshin Aug 8, 2018
f62fe43
[SPARK-25036][SQL][FOLLOW-UP] Avoid match may not be exhaustive in Sc…
ueshin Aug 8, 2018
a40806d
[SPARK-23596][SQL] Test interpreted path on encoders test suites
viirya Aug 9, 2018
519e03d
[SPARK-25058][SQL] Use Block.isEmpty/nonEmpty to check whether the co…
ueshin Aug 9, 2018
56e9e97
[MINOR][DOC] Fix typo
kiszk Aug 9, 2018
386fbd3
[SPARK-23415][SQL][TEST] Make behavior of BufferHolderSparkSubmitSuit…
kiszk Aug 9, 2018
b2950ce
Revert "[SPARK-24648][SQL] SqlMetrics should be threadsafe"
cloud-fan Aug 9, 2018
1a7e747
[SPARK-25047][ML] Can't assign SerializedLambda to scala.Function1 in…
srowen Aug 9, 2018
2949a83
[SPARK-25063][SQL] Rename class KnowNotNull to KnownNotNull
maryannxue Aug 9, 2018
d365397
[SPARK-24626][SQL] Improve location size calculation in Analyze Table…
Achuth17 Aug 9, 2018
eb9a696
[MINOR][BUILD] Update Jetty to 9.3.24.v20180605
srowen Aug 9, 2018
bd6db15
[SPARK-25077][SQL] Delete unused variable in WindowExec
Aug 9, 2018
fec67ed
[SPARK-25076][SQL] SQLConf should not be retrieved from a stopped Spa…
cloud-fan Aug 9, 2018
9b8521e
[SPARK-25068][SQL] Add exists function.
ueshin Aug 9, 2018
6c7bb57
[SPARK-24886][INFRA] Fix the testing script to increase timeout for J…
HyukjinKwon Aug 10, 2018
bdd2796
[SPARK-24251][SQL] Add analysis tests for AppendData.
rdblue Aug 10, 2018
0cea9e3
[SPARK-24855][SQL][EXTERNAL] Built-in AVRO support should support spe…
lindblombr Aug 10, 2018
ab1029f
[SPARK-23912][SQL][FOLLOWUP] Refactor ArrayDistinct
kiszk Aug 10, 2018
9abe09b
[SPARK-24127][SS] Continuous text socket source
arunmahadevan Aug 10, 2018
4f17585
[SPARK-19355][SQL] Use map output statistics to improve global limit'…
viirya Aug 10, 2018
132bcce
[SPARK-25036][SQL] Avoid discarding unmoored doc comment in Scala-2.12.
kiszk Aug 10, 2018
1dd0f17
[SPARK-25036][SQL][FOLLOW-UP] Avoid match may not be exhaustive in Sc…
kiszk Aug 10, 2018
91cdab5
[MINOR][BUILD] Add ECCN notice required by http://www.apache.org/dev/…
srowen Aug 10, 2018
f5aba65
[SPARK-25081][CORE] Nested spill in ShuffleExternalSorter should not …
zsxwing Aug 10, 2018
4b11d90
[MINOR][DOC] Add missing compression codec .
10110346 Aug 11, 2018
b73eb0e
[MINOR][DOC] Add missing compression codec .
10110346 Aug 11, 2018
41a7de6
[SPARK-25084][SQL] "distribute by" on multiple columns (wrap in brack…
yucai Aug 11, 2018
4855d5c
[SPARK-24822][PYSPARK] Python support for barrier execution mode
jiangxb1987 Aug 11, 2018
8ec25cd
Fix typos detected by github.com/client9/misspell
seratch Aug 12, 2018
c3be2cd
[SPARK-25092] Add RewriteExceptAll and RewriteIntersectAll in the lis…
dilipbiswal Aug 12, 2018
d177234
[SQL][TEST][MINOR] Add missing codes to ParquetCompressionCodecPreced…
kiszk Aug 12, 2018
5bc7598
Fix typos
tynan-cr Aug 12, 2018
a90b1f5
[MINOR][DOC] Fix Java example code in Column's comments
Aug 12, 2018
be2238f
[SPARK-24774][SQL] Avro: Support logical decimal type
gengliangwang Aug 13, 2018
02d0a1f
[SPARK-25069][CORE] Using UnsafeAlignedOffset to make the entire reco…
eatoncys Aug 13, 2018
20fa456
[SPARK-25090][ML] Enforce implicit type coercion in ParamGridBuilder
mgaido91 Aug 13, 2018
5d6abad
[SPARK-25033] Bump Apache commons.{httpclient, httpcore}
Aug 13, 2018
a992827
[SPARK-24420][BUILD][FOLLOW-UP] Upgrade ASM6 APIs
gatorsmile Aug 13, 2018
b270bcc
[SPARK-25096][SQL] Loosen nullability if the cast is force-nullable.
ueshin Aug 13, 2018
ab06c25
[SPARK-24391][SQL] Support arrays of any types by from_json
MaxGekk Aug 13, 2018
26775e3
[SPARK-25099][SQL][TEST] Generate Avro Binary files in test suite
gengliangwang Aug 13, 2018
2e3abdf
[SPARK-22713][CORE] ExternalAppendOnlyMap leaks when spilled during i…
Aug 13, 2018
b804ca5
[SPARK-23908][SQL][FOLLOW-UP] Rename inputs to arguments, and add arg…
ueshin Aug 13, 2018
c220cc4
[SPARK-25028][SQL] Avoid NPE when analyzing partition with NULL values
mgaido91 Aug 13, 2018
ab19730
[SPARK-25104][SQL] Avro: Validate user specified output schema
gengliangwang Aug 14, 2018
3eb5209
[SPARK-22974][ML] Attach attributes to output column of CountVectorModel
viirya Aug 14, 2018
e2ab7de
[MINOR][SQL][DOC] Fix `to_json` example in function description and doc
dongjoon-hyun Aug 14, 2018
42263fd
[SPARK-23938][SQL] Add map_zip_with function
mn-mikke Aug 14, 2018
80784a1
[SPARK-18057][FOLLOW-UP] Use 127.0.0.1 to avoid zookeeper picking up …
zsxwing Aug 14, 2018
1024875
[SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults.
squito Aug 14, 2018
b81e303
[SPARK-25043] print master and appId from spark-sql on startup
abellina Aug 14, 2018
3c614d0
[SPARK-25113][SQL] Add logging to CodeGenerator when any generated me…
rednaxelafx Aug 14, 2018
92fd7f3
[SPARK-25115][CORE] Eliminate extra memory copy done when a ByteBuf i…
normanmaurer Aug 15, 2018
ed075e1
[SPARK-23874][SQL][PYTHON] Upgrade Apache Arrow to 0.10.0
BryanCutler Aug 15, 2018
19c45db
[SPARK-24505][SQL] Convert strings in codegen to blocks: Cast and Bou…
viirya Aug 15, 2018
4d8ae0d
[SPARK-25111][BUILD] increment kinesis client/producer & aws-sdk vers…
steveloughran Aug 15, 2018
bfb7439
[SPARK-24819][CORE] Fail fast when no enough slots to launch the barr…
jiangxb1987 Aug 15, 2018
717f58e
[SPARK-24685][BUILD] Restore support for building old Hadoop versions…
Aug 15, 2018
a791c29
[SPARK-23984][K8S] Changed Python Version config to be camelCase
ifilonenko Aug 16, 2018
7822c3f
[SPARK-25082][SQL] improve the javadoc for expm1()
Aug 16, 2018
5b4a38d
[SPARK-23939][SQL] Add transform_keys function
codeatri Aug 16, 2018
ea63a7a
[SPARK-23932][SQL] Higher order function zip_with
techaddict Aug 16, 2018
b3e6fe7
[SPARK-23654][BUILD] remove jets3t as a dependency of spark
srowen Aug 16, 2018
e501924
[SPARK-24555][ML] logNumExamples in KMeans/BiKM/GMM/AFT/NB
zhengruifeng Aug 16, 2018
e59dd8f
[SPARK-25092][SQL][FOLLOWUP] Add RewriteCorrelatedScalarSubquery in l…
dilipbiswal Aug 16, 2018
709f541
[DOCS] Update configuration.md
KraFusion Aug 16, 2018
30be71e
[DOCS] Fix cloud-integration.md Typo
KraFusion Aug 16, 2018
9251c61
[SPARK-24665][PYSPARK][FOLLOWUP] Use SQLConf in PySpark to manage all…
xuanyuanking Aug 17, 2018
f161409
[SPARK-23940][SQL] Add transform_values SQL function
codeatri Aug 17, 2018
8af61fb
[SPARK-25122][SQL] Deduplication of supports equals code
mn-mikke Aug 17, 2018
c1ffb3c
[SPARK-23938][SQL][FOLLOW-UP][TEST] Nullabilities of value arguments …
ueshin Aug 17, 2018
162326c
[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in R
dilipbiswal Aug 17, 2018
8b0e94d
[SPARK-23042][ML] Use OneHotEncoderModel to encode labels in Multilay…
viirya Aug 17, 2018
da2dc69
[SPARK-25116][TESTS] Fix the Kafka cluster leak and clean up cached p…
zsxwing Aug 17, 2018
ba84bcb
[SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
ifilonenko Aug 17, 2018
10f2b6f
[SPARK-23555][PYTHON] Add BinaryType support for Arrow in Python
BryanCutler Aug 18, 2018
e3cf13d
[SPARK-25137][SPARK SHELL] NumberFormatException` when starting spark…
vinodkc Aug 18, 2018
f454d52
[MINOR][DOC][SQL] use one line for annotation arg value
mengxr Aug 18, 2018
4dd87d8
[SPARK-25142][PYSPARK] Add error messages when Python worker could no…
ueshin Aug 18, 2018
9047cc0
[SPARK-24886][INFRA] Fix the testing script to increase timeout for J…
HyukjinKwon Aug 18, 2018
14d7c1c
[SPARK-24863][SS] Report Kafka offset lag as a custom metrics
arunmahadevan Aug 18, 2018
a8a1ac0
[SPARK-24959][SQL] Speed up count() for JSON and CSV
MaxGekk Aug 18, 2018
6b8fbbf
[SPARK-25141][SQL][TEST] Modify tests for higher-order functions to c…
ueshin Aug 19, 2018
60af250
[SPARK-25160][SQL] Avro: remove sql configuration spark.sql.avro.outp…
gengliangwang Aug 20, 2018
219ed7b
[DOCS] Fixed NDCG formula issues
yueguoguo Aug 20, 2018
883f3af
[SPARK-25144][SQL][TEST] Free aggregate map when task ends
dongjoon-hyun Aug 21, 2018
b461acb
[SPARK-25134][SQL] Csv column pruning with checking of headers throws…
koertkuipers Aug 21, 2018
f984ec7
[SPARK-25132][SQL] Case-insensitive field resolution when reading fro…
seancxmao Aug 21, 2018
4fb96e5
[SPARK-25114][CORE] Fix RecordBinaryComparator when subtraction betwe…
jiangxb1987 Aug 21, 2018
b8788b3
[BUILD] Close stale PRs
srowen Aug 21, 2018
5059255
[SPARK-25161][CORE] Fix several bugs in failure handling of barrier e…
jiangxb1987 Aug 21, 2018
d800632
[MINOR] Add .crc files to .gitignore
HeartSaVioR Aug 21, 2018
35f7f5c
[DOCS][MINOR] Fix a few broken links and typos, and, nit, use HTTPS m…
srowen Aug 21, 2018
99d2e4e
[SPARK-24296][CORE] Replicate large blocks as a stream.
squito Aug 21, 2018
72ecfd0
[SPARK-25149][GRAPHX] Update Parallel Personalized Page Rank to test …
MrBago Aug 21, 2018
6c5cb85
[SPARK-24763][SS] Remove redundant key data from value in streaming a…
HeartSaVioR Aug 21, 2018
ac0174e
[SPARK-25129][SQL] Make the mapping of com.databricks.spark.avro to b…
gengliangwang Aug 21, 2018
42035a4
[SPARK-24441][SS] Expose total estimated size of states in HDFSBacked…
HeartSaVioR Aug 21, 2018
ad45299
[SPARK-25095][PYSPARK] Python support for BarrierTaskContext
jiangxb1987 Aug 21, 2018
a998e9d
[MINOR] Added import to fix compilation
tdas Aug 22, 2018
07737c8
[SPARK-23711][SPARK-25140][SQL] Catch correct exceptions when expr co…
maropu Aug 22, 2018
4a9c9d8
[SPARK-25159][SQL] json schema inference should only trigger one job
cloud-fan Aug 22, 2018
55f3664
[SPARK-25093][SQL] Avoid recompiling regexp for comments multiple times
mgaido91 Aug 22, 2018
e754887
[SPARK-24882][SQL] improve data source v2 API
cloud-fan Aug 22, 2018
71f38ac
[SPARK-23698][PYTHON] Resolve undefined names in Python 3
Aug 22, 2018
2381953
[SPARK-25105][PYSPARK][SQL] Include PandasUDFType in the import all o…
kevinyu98 Aug 22, 2018
68ec4d6
[SPARK-25181][CORE] Limit Thread Pool size in BlockManager Master and…
mukulmurthy Aug 22, 2018
3106324
[SPARK-25184][SS] Fixed race condition in StreamExecution that caused…
tdas Aug 22, 2018
49a1993
[SPARK-25163][SQL] Fix flaky test: o.a.s.util.collection.ExternalAppe…
viirya Aug 22, 2018
2bc7b75
[SPARK-24785][SHELL] Making sure REPL prints Spark UI info and then W…
dbtsai Aug 22, 2018
0295ad4
[SPARK-25127] DataSourceV2: Remove SupportsPushDownCatalystFilters
rxin Aug 23, 2018
1747469
[SPARK-25167][SPARKR][TEST][MINOR] Minor fixes for R sql tests
dilipbiswal Aug 23, 2018
05974f9
[SPARK-25133][SQL][DOC] Avro data source guide
gengliangwang Aug 23, 2018
4972090
[SPARK-23932][SQL][FOLLOW-UP] Fix an example of zip_with function.
ueshin Aug 23, 2018
2a0a8f7
[SPARK-23034][SQL] Show RDD/relation names in RDD/Hive table scan nodes
maropu Aug 23, 2018
8cc591c
[SPARK-25164][SQL] Avoid rebuilding column and path list for each col…
bersprockets Aug 23, 2018
e3b7bb4
[SPARK-24811][FOLLOWUP][SQL] Revise package of AvroDataToCatalyst and…
gengliangwang Aug 23, 2018
5d572fc
[SPARK-25126][SQL] Avoid creating Reader for all orc files
Aug 23, 2018
a9aacdf
[SPARK-25208][SQL] Loosen Cast.forceNullable for DecimalType.
ueshin Aug 23, 2018
8ed0449
[SPARK-25204][SS] Fix race in rate source test.
jose-torres Aug 23, 2018
b5e1188
[SPARK-25124][ML] VectorSizeHint setSize and getSize don't return values
huaxingao Aug 23, 2018
0ce09ec
[SPARK-25205][CORE] Fix typo in spark.network.crypto.keyFactoryIterat…
squito Aug 24, 2018
b88ddb8
[SPARK-23425][SQL] Support wildcard in HDFS path for load table command
sujith71955 Aug 24, 2018
cd6dff7
[SPARK-25209][SQL] Avoid deserializer check in Dataset.apply when Dat…
bogdanrdc Aug 24, 2018
f2d3542
[SPARK-4502][SQL] Parquet nested column pruning - foundation
mallman Aug 24, 2018
9b6baeb
[SPARK-25029][BUILD][CORE] Janino "Two non-abstract methods ..." errors
srowen Aug 24, 2018
ab33028
[SPARK-25178][SQL] Directly ship the StructType objects of the keySch…
kiszk Aug 24, 2018
c20916a
[SPARK-25073][YARN] AM and Executor Memory validation message is not …
sujith71955 Aug 24, 2018
8bb9414
[SPARK-25214][SS] Fix the issue that Kafka v2 source may return dupli…
zsxwing Aug 24, 2018
f8346d2
[SPARK-25174][YARN] Limit the size of diagnostic message for am to un…
yaooqinn Aug 24, 2018
9714fa5
[SPARK-25234][SPARKR] avoid integer overflow in parallelize
mengxr Aug 24, 2018
8e64278
Correct missing punctuation in the documentation
Aug 25, 2018
3e4f166
[MINOR] Fix Scala 2.12 build
dbtsai Aug 25, 2018
6c66ab8
[SPARK-24688][EXAMPLES] Modify the comments about LabeledPoint
huangweizhe123 Aug 25, 2018
c17a8ff
[SPARK-25214][SS][FOLLOWUP] Fix the issue that Kafka v2 source may re…
zsxwing Aug 25, 2018
ad43e2c
[SPARK-23792][DOCS] Documentation improvements for datetime functions
abradbury Aug 26, 2018
5cdb8a2
[SPARK-23698][PYTHON][FOLLOWUP] Resolve undefiend names in setup.py
HyukjinKwon Aug 27, 2018
5c27b0d
[SPARK-19355][SQL][FOLLOWUP] Remove the child.outputOrdering check in…
viirya Aug 27, 2018
6193a20
[SPARK-24978][SQL] Add spark.sql.fast.hash.aggregate.row.max.capacity…
heary-cao Aug 27, 2018
381a967
[SPARK-25249][CORE][TEST] add a unit test for OpenHashMap
10110346 Aug 27, 2018
810d59c
[SPARK-24882][FOLLOWUP] Fix flaky synchronization in Kafka tests.
jose-torres Aug 27, 2018
c3f285c
[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the …
wangyum Aug 27, 2018
dac099d
[SPARK-24090][K8S] Update running-on-kubernetes.md
liyinan926 Aug 27, 2018
8198ea5
[SPARK-24721][SQL] Exclude Python UDFs filters in FileSourceStrategy
icexelloss Aug 28, 2018
592e3a4
[SPARK-25218][CORE] Fix potential resource leaks in TransportServer a…
zsxwing Aug 28, 2018
1149c4e
[SPARK-25005][SS] Support non-consecutive offsets for Kafka
zsxwing Aug 28, 2018
de46df5
[SPARK-23997][SQL] Configurable maximum number of buckets
ferdonline Aug 28, 2018
4e3f3ce
[SPARK-23679][YARN] Setting RM_HA_URLS for AmIpFilter to avoid redire…
jerryshao Aug 28, 2018
aff8f15
[SPARK-25240][SQL] Fix for a deadlock in RECOVER PARTITIONS
MaxGekk Aug 28, 2018
7ad18ee
[SPARK-25004][CORE] Add spark.executor.pyspark.memory limit.
rdblue Aug 28, 2018
1038540
[SPARK-25212][SQL] Support Filter in ConvertToLocalRelation
bogdanrdc Aug 28, 2018
bbbf814
[SPARK-22357][CORE] SparkContext.binaryFiles ignore minPartitions par…
bomeng Aug 29, 2018
32c8a3d
[MINOR] Avoid code duplication for nullable in Higher Order function
mgaido91 Aug 29, 2018
68ec207
[SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType
arunmahadevan Aug 29, 2018
38391c9
[SPARK-25253][PYSPARK] Refactor local connection & auth code
squito Aug 29, 2018
ff8dcc1
[SPARK-25235][SHELL] Merge the REPL code in Scala 2.11 and 2.12 branches
dbtsai Aug 29, 2018
82c18c2
[SPARK-23030][SQL][PYTHON] Use Arrow stream format for creating from …
BryanCutler Aug 29, 2018
1fd59c1
[WIP][SPARK-25044][SQL] (take 2) Address translation of LMF closure …
srowen Aug 29, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[MINOR] Avoid code duplication for nullable in Higher Order function
## What changes were proposed in this pull request?

Most of  `HigherOrderFunction`s have the same `nullable` definition, ie. they are nullable when one of their arguments is nullable. The PR refactors it in order to avoid code duplication.

## How was this patch tested?

NA

Closes apache#22243 from mgaido91/MINOR_nullable_hof.

Authored-by: Marco Gaido <[email protected]>
Signed-off-by: hyukjinkwon <[email protected]>
  • Loading branch information
mgaido91 authored and HyukjinKwon committed Aug 29, 2018
commit 32c8a3d7beac4b47a75f5ec3c69b13ebc57de0c7
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ object LambdaFunction {
*/
trait HigherOrderFunction extends Expression with ExpectsInputTypes {

override def nullable: Boolean = arguments.exists(_.nullable)

override def children: Seq[Expression] = arguments ++ functions

/**
Expand Down Expand Up @@ -217,8 +219,6 @@ case class ArrayTransform(
function: Expression)
extends ArrayBasedSimpleHigherOrderFunction with CodegenFallback {

override def nullable: Boolean = argument.nullable

override def dataType: ArrayType = ArrayType(function.dataType, function.nullable)

override def bind(f: (Expression, Seq[(DataType, Boolean)]) => LambdaFunction): ArrayTransform = {
Expand Down Expand Up @@ -287,8 +287,6 @@ case class MapFilter(
copy(function = f(function, (keyType, false) :: (valueType, valueContainsNull) :: Nil))
}

override def nullable: Boolean = argument.nullable

override def nullSafeEval(inputRow: InternalRow, argumentValue: Any): Any = {
val m = argumentValue.asInstanceOf[MapData]
val f = functionForEval
Expand Down Expand Up @@ -328,8 +326,6 @@ case class ArrayFilter(
function: Expression)
extends ArrayBasedSimpleHigherOrderFunction with CodegenFallback {

override def nullable: Boolean = argument.nullable

override def dataType: DataType = argument.dataType

override def functionType: AbstractDataType = BooleanType
Expand Down Expand Up @@ -375,8 +371,6 @@ case class ArrayExists(
function: Expression)
extends ArrayBasedSimpleHigherOrderFunction with CodegenFallback {

override def nullable: Boolean = argument.nullable

override def dataType: DataType = BooleanType

override def functionType: AbstractDataType = BooleanType
Expand Down Expand Up @@ -516,8 +510,6 @@ case class TransformKeys(
function: Expression)
extends MapBasedSimpleHigherOrderFunction with CodegenFallback {

override def nullable: Boolean = argument.nullable

@transient lazy val MapType(keyType, valueType, valueContainsNull) = argument.dataType

override def dataType: DataType = MapType(function.dataType, valueType, valueContainsNull)
Expand Down Expand Up @@ -568,8 +560,6 @@ case class TransformValues(
function: Expression)
extends MapBasedSimpleHigherOrderFunction with CodegenFallback {

override def nullable: Boolean = argument.nullable

@transient lazy val MapType(keyType, valueType, valueContainsNull) = argument.dataType

override def dataType: DataType = MapType(keyType, function.dataType, function.nullable)
Expand Down Expand Up @@ -638,8 +628,6 @@ case class MapZipWith(left: Expression, right: Expression, function: Expression)

override def functionTypes: Seq[AbstractDataType] = AnyDataType :: Nil

override def nullable: Boolean = left.nullable || right.nullable

override def dataType: DataType = MapType(keyType, function.dataType, function.nullable)

override def bind(f: (Expression, Seq[(DataType, Boolean)]) => LambdaFunction): MapZipWith = {
Expand Down Expand Up @@ -810,8 +798,6 @@ case class ZipWith(left: Expression, right: Expression, function: Expression)

override def functionTypes: Seq[AbstractDataType] = AnyDataType :: Nil

override def nullable: Boolean = left.nullable || right.nullable

override def dataType: ArrayType = ArrayType(function.dataType, function.nullable)

override def bind(f: (Expression, Seq[(DataType, Boolean)]) => LambdaFunction): ZipWith = {
Expand Down