Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
9a6658c
[SPARK-17477]: SparkSQL cannot handle schema evolution from Int -> Lo…
wgtmac Sep 9, 2016
d978d4b
[SPARK-17354] [SQL] Partitioning by dates/timestamps should work with…
HyukjinKwon Sep 9, 2016
965d966
[SPARK-15453][SQL] FileSourceScanExec to extract `outputOrdering` inf…
tejasapatil Sep 10, 2016
a30257b
[SPARK-11496][GRAPHX] Parallel implementation of personalized pagerank
Sep 10, 2016
bca8f30
[SPARK-15509][FOLLOW-UP][ML][SPARKR] R MLlib algorithms should suppor…
yanboliang Sep 10, 2016
705fbdc
[SPARK-17396][CORE] Share the task support between UnionRDD instances.
rdblue Sep 10, 2016
9e6680e
[SPARK-16445][MLLIB][SPARKR] Fix @return description for sparkR mlp s…
keypointt Sep 10, 2016
71d6291
[SPARK-17389][ML][MLLIB] KMeans speedup with better choice of k-means…
srowen Sep 11, 2016
eb10a06
[SPARK-17439][SQL] Fixing compression issues with approximate quantil…
thunterdb Sep 11, 2016
8ce0d5e
[SPARK-17330][SPARK UT] Clean up spark-warehouse in UT
Sep 11, 2016
1d4165f
[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH fro…
BryanCutler Sep 11, 2016
7d309ff
[SPARK-17389][FOLLOW-UP][ML] Change KMeans k-means|| default init ste…
yanboliang Sep 11, 2016
ed112bd
[SPARK-17415][SQL] Better error message for driver-side broadcast joi…
sameeragarwal Sep 11, 2016
7afab77
[SPARK-17486] Remove unused TaskMetricsUIData.updatedBlockStatuses field
JoshRosen Sep 12, 2016
d608f99
[SPARK-17171][WEB UI] DAG will list all partitions in the graph
cenyuhai Sep 12, 2016
10b1feb
[SPARK-17447] Performance improvement in Partitioner.defaultPartition…
codlife Sep 12, 2016
28f6379
[SPARK-16992][PYSPARK] use map comprehension in doc
gsemet Sep 12, 2016
7037d03
[SPARK CORE][MINOR] fix "default partitioner cannot partition array k…
WeichenXu123 Sep 12, 2016
cb6e59c
[SPARK-17503][CORE] Fix memory leak in Memory store when unable to ca…
clockfly Sep 12, 2016
1984289
[SPARK-17483] Refactoring in BlockManager status reporting and block …
JoshRosen Sep 12, 2016
1c60199
[SPARK-14818] Post-2.0 MiMa exclusion and build changes
JoshRosen Sep 12, 2016
671d8ec
[SPARK-17485] Prevent failed remote reads of cached blocks from faili…
JoshRosen Sep 12, 2016
77808e0
[SPARK-17474] [SQL] fix python udf in TakeOrderedAndProjectExec
Sep 12, 2016
67b0a5e
[SPARK-17515] CollectLimit.execute() should perform per-partition limits
JoshRosen Sep 13, 2016
2dda3bb
[SPARK-17142][SQL] Complex query triggers binding error in HashAggreg…
jiangxb1987 Sep 13, 2016
33fd9a7
[SPARK-17531] Don't initialize Hive Listeners for the Execution Client
brkyvz Sep 13, 2016
d020ea3
[SPARK-17530][SQL] Add Statistics into DESCRIBE FORMATTED
gatorsmile Sep 13, 2016
9b48546
[SPARK-17317][SPARKR] Add SparkR vignette
junyangq Sep 14, 2016
8cbdf79
[SPARK-17449][DOCUMENTATION] Relation between heartbeatInterval and…
jagadeesanas2 Sep 14, 2016
7cd75c0
[SPARK-17525][PYTHON] Remove SparkContext.clearFiles() from the PySpa…
sjakthol Sep 14, 2016
c37fda6
[CORE][DOC] remove redundant comment
wangmiao1981 Sep 14, 2016
84dced4
[SPARK-17480][SQL] Improve performance by removing or caching List.le…
seyfe Sep 14, 2016
25ca2de
[SPARK-17445][DOCS] Reference an ASF page as the main place to find t…
srowen Sep 14, 2016
2125957
[SPARK-17409][SQL] Do Not Optimize Query in CTAS More Than Once
gatorsmile Sep 14, 2016
f9f2c6b
[SPARK-17514] df.take(1) and df.limit(1).collect() should perform the…
JoshRosen Sep 14, 2016
08b9c75
[MINOR][SQL] Add missing functions for some options in SQLConf and us…
HyukjinKwon Sep 14, 2016
a1d246e
[SPARK-10747][SQL] Support NULLS FIRST|LAST clause in ORDER BY
xwu0226 Sep 14, 2016
a63556f
[SPARK-17511] Yarn Dynamic Allocation: Avoid marking released contain…
kishorvpatil Sep 14, 2016
e5814cc
[SPARK-17463][CORE] Make CollectionAccumulator and SetAccumulator's v…
zsxwing Sep 14, 2016
3b80d1f
[SPARK-17472] [PYSPARK] Better error message for serialization failur…
ericl Sep 14, 2016
795d83e
[SPARK-17465][SPARK CORE] Inappropriate memory management in `org.apa…
Sep 14, 2016
f1a0223
[SPARK-17440][SPARK-17441] Fixed Multiple Bugs in ALTER TABLE
gatorsmile Sep 15, 2016
de885a8
[SPARK-17507][ML][MLLIB] check weight vector size in ANN
WeichenXu123 Sep 15, 2016
5ee52aa
[SPARK-17524][TESTS] Use specified spark.buffer.pageSize
a-roberts Sep 15, 2016
037565f
[SPARK-17521] Error when I use sparkContext.makeRDD(Seq())
codlife Sep 15, 2016
6f0e760
[SPARK-17406][WEB UI] limit timeline executor events
cenyuhai Sep 15, 2016
74bf9a2
[SPARK-17536][SQL] Minor performance improvement to JDBC batch inserts
Sep 15, 2016
396a6ce
[SPARK-17406][BUILD][HOTFIX] MiMa excludes fix
srowen Sep 15, 2016
4b29340
[SPARK-17451][CORE] CoarseGrainedExecutorBackend should inform driver…
tejasapatil Sep 15, 2016
1430e3b
[SPARK-17379][BUILD] Upgrade netty-all to 4.0.41 final for bug fixes
a-roberts Sep 15, 2016
129e87b
[SPARK-17547] Ensure temp shuffle data file is cleaned up after error
JoshRosen Sep 15, 2016
be726e2
[SPARK-17114][SQL] Fix aggregates grouped by literals with empty input
hvanhovell Sep 15, 2016
17f7ce3
[SPARK-17429][SQL] use ImplicitCastInputTypes with function Length
cenyuhai Sep 15, 2016
609e6ce
[SPARK-17364][SQL] Antlr lexer wrongly treats full qualified identifi…
clockfly Sep 15, 2016
2dc869f
[SPARK-17484] Prevent invalid block locations from being reported aft…
JoshRosen Sep 15, 2016
2bb1a19
[SPARK-17458][SQL] Alias specified for aggregates in a pivot are not …
aray Sep 15, 2016
8098b29
[SPARK-17543] Missing log4j config file for tests in common/network-…
jagadeesanas2 Sep 16, 2016
fdb9154
[SPARK-17534][TESTS] Increase timeouts for DirectKafkaStreamSuite tests
a-roberts Sep 16, 2016
0ef4313
[SPARK-17426][SQL] Refactor `TreeNode.toJSON` to avoid OOM when conve…
clockfly Sep 16, 2016
ef7fa83
[SPARK-17558] Bump Hadoop 2.7 version from 2.7.2 to 2.7.3
rxin Sep 16, 2016
ccff86d
[SPARK-17561][DOCS] DataFrameWriter documentation formatting problems
srowen Sep 16, 2016
60c287f
[SPARK-17549][SQL] Only collect table size stat in driver for cached …
Sep 16, 2016
19f894e
Correct fetchsize property name in docs
darabos Sep 17, 2016
ae3b76b
[SPARK-17567][DOCS] Use valid url to Spark RDD paper
keypointt Sep 17, 2016
b7d1923
[SPARK-17548][MLLIB] Word2VecModel.findSynonyms no longer spuriously …
willb Sep 17, 2016
06684bb
[SPARK-17529][CORE] Implement BitSet.clearUntil and use it during mer…
Sep 17, 2016
19132d5
[SPARK-17575][DOCS] Remove extra table tags in configuration document
phalodi Sep 17, 2016
defe9aa
[SPARK-17480][SQL][FOLLOWUP] Fix more instances which calls List.leng…
HyukjinKwon Sep 17, 2016
21035a6
[SPARK-17491] Close serialization stream to fix wrong answer bug in p…
JoshRosen Sep 17, 2016
d6cbf8d
[SPARK-17518][SQL] Block Users to Specify the Internal Data Source Pr…
gatorsmile Sep 18, 2016
3357bc7
[SPARK-17541][SQL] fix some DDL bugs about table management when same…
cloud-fan Sep 18, 2016
7d24523
[SPARK-17506][SQL] Improve the check double values equality rule.
jiangxb1987 Sep 18, 2016
e460dcd
[SPARK-17546][DEPLOY] start-* scripts should use hostname -f
srowen Sep 18, 2016
032335e
[SPARK-17586][BUILD] Do not call static member via instance reference
HyukjinKwon Sep 18, 2016
5930e0b
[SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV cast null value…
lw-lin Sep 18, 2016
98dfd17
[SPARK-17571][SQL] AssertOnQuery.condition should always return Boole…
petermaxlee Sep 18, 2016
3b18214
[SPARK-17297][DOCS] Clarify window/slide duration as absolute time, n…
srowen Sep 19, 2016
2b7ebff
[SPARK-17473][SQL] fixing docker integration tests error due to diffe…
sureshthalamati Sep 19, 2016
e6a9a72
[SPARK-17438][WEBUI] Show Application.executorLimit in the applicatio…
zsxwing Sep 19, 2016
f6aeaf2
revert change
wgtmac Sep 19, 2016
9fc18a4
[SPARK-17477][SQL] SparkSQL cannot handle schema evolution from Int -…
wgtmac Sep 19, 2016
8efd4dd
[SPARK-16439] [SQL] bring back the separator in SQL UI
Sep 19, 2016
a1bdea0
[SPARK-17100] [SQL] fix Python udf in filter on top of outer join
Sep 19, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-17477][SQL] SparkSQL cannot handle schema evolution from Int -…
…> Long when parquet files have Int as its type while hive metastore has Long as its type
  • Loading branch information
wgtmac committed Sep 19, 2016
commit 9fc18a419633a24fc90acaf82378a46404671b34
Original file line number Diff line number Diff line change
Expand Up @@ -214,9 +214,25 @@ private[parquet] class ParquetRowConverter(
updater: ParentContainerUpdater): Converter with HasParentContainerUpdater = {

catalystType match {
case BooleanType | IntegerType | LongType | FloatType | DoubleType | BinaryType =>
case BooleanType | IntegerType | FloatType | DoubleType | BinaryType =>
new ParquetPrimitiveConverter(updater)

/**
* When reading a hive table of parquet files with schema evolution from
* Int to Long, if hive metastore has Long as its type while parquet files
* have Int, SparkSQL need to differentiate the actual type in the parquet
* files. Otherwise, it will result in java.lang.ClassCastException:
* [[MutableLong]] cannot be cast to [[MutableInt]].
*/
case LongType if parquetType == INT64 =>
new ParquetPrimitiveConverter(updater)

case LongType =>
new ParquetPrimitiveConverter(updater) {
override def addInt(value: Int): Unit =
updater.setLong(value.asInstanceOf[Long])
}

case ByteType =>
new ParquetPrimitiveConverter(updater) {
override def addInt(value: Int): Unit =
Expand Down