Branch 1.6#13882

Closed

liu549676915 wants to merge 916 commits intomasterfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 11, 2016

[SPARK-12734][HOTFIX] Build changes must trigger all tests; clean after install in dep tests
JoshRosen
committed
[SPARK-12758][SQL] add note to Spark SQL Migration guide about TimestampType casting

blbradley
authored and
marmbrus
committed

Commits on Jan 12, 2016

[SPARK-11823] Ignores HiveThriftBinaryServerSuite's test jdbc cancel

yhuai
authored and
JoshRosen
committed
[SPARK-12638][API DOC] Parameter explanation not very accurate for rdd function "aggregate"

Wenpei
authored and
srowen
committed
[SPARK-12582][TEST] IndexShuffleBlockResolverSuite fails in windows

Yucai Yu
authored and
srowen
committed
[SPARK-5273][MLLIB][DOCS] Improve documentation examples for LinearRegression
srowen
committed
[SPARK-7615][MLLIB] MLLIB Word2Vec wordVectors divided by Euclidean Norm equals to zero
srowen
committed
Revert "[SPARK-12645][SPARKR] SparkR support hash function"
yhuai
committed

Commits on Jan 13, 2016

[HOT-FIX] bypass hive test when parse logical plan to json

cloud-fan
authored and
marmbrus
committed
[SPARK-12558][SQL] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12805][MESOS] Fixes documentation on Mesos run modes

Luc Bourlier
authored and
rxin
committed
[SPARK-12685][MLLIB][BACKPORT TO 1.4] word2vec trainWordsCount gets overflow

hhbyyh
authored and
jkbradley
committed
[SPARK-12268][PYSPARK] Make pyspark shell pythonstartup work under python3

erikselin
authored and
JoshRosen
committed
[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

carsonwang
authored and
JoshRosen
committed

Commits on Jan 14, 2016

Commits on Jan 15, 2016

[SPARK-12708][UI] Sorting task error in Stages Page when yarn mode.

yoshidakuy
authored and
sarutak
committed
[SPARK-11031][SPARKR] Method str() on a DataFrame

Oscar D. Lara Yejas
authored and
shivaram
committed
[SPARK-12701][CORE] FileAppender should use join to ensure writing thread completion

BryanCutler
authored and
srowen
committed

Commits on Jan 16, 2016

[SPARK-12722][DOCS] Fixed typo in Pipeline example

Jeff Lam
authored and
srowen
committed

Commits on Jan 18, 2016

[SPARK-12558][FOLLOW-UP] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12346][ML] Missing attribute names in GLM for vector-type features

ericl
authored and
mengxr
committed
[SPARK-12814][DOCUMENT] Add deploy instructions for Python in flume integration doc

zsxwing
authored and
tdas
committed

Commits on Jan 19, 2016

Commits on Jan 21, 2016

[SPARK-12921] Use SparkHadoopUtil reflection in SpecificParquetRecordReaderBase
JoshRosen
committed

Commits on Jan 22, 2016

[SPARK-12747][SQL] Use correct type name for Postgres JDBC's real array

viirya
authored and
rxin
committed

Commits on Jan 23, 2016

Commits on Jan 24, 2016

[SPARK-12120][PYSPARK] Improve exception message when failing to init…

zjffdu
authored and
JoshRosen
committed

Commits on Jan 25, 2016

[SPARK-12624][PYSPARK] Checks row length when converting Java arrays to Python rows

liancheng
authored and
yhuai
committed
[SPARK-12932][JAVA API] improved error message for java type inference failure

andygrove
authored and
srowen
committed
[SPARK-12755][CORE] Stop the event logger before the DAG scheduler

Michael Allman
authored and
srowen
committed

Commits on Jan 26, 2016

Commits on Jan 27, 2016

Commits on Jan 29, 2016

[SPARK-13082][PYSPARK] Backport the fix of 'read.json(rdd)' in #10559 to branch-1.6
zsxwing
committed

Commits on Jan 30, 2016

[SPARK-13088] Fix DAG viz in latest version of chrome
Andrew Or
committed

Commits on Feb 1, 2016

Commits on Feb 2, 2016

[SPARK-13087][SQL] Fix group by function for sort based aggregation

marmbrus
authored and
yhuai
committed
[SPARK-13094][SQL] Add encoders for seq/array of primitives
marmbrus
committed
[SPARK-12780][ML][PYTHON][BACKPORT] Inconsistency returning value of ML python models' properties

yinxusen
authored and
jkbradley
committed
[SPARK-12629][SPARKR] Fixes for DataFrame saveAsTable method

NarineK
authored and
shivaram
committed
[SPARK-13121][STREAMING] java mapWithState mishandles scala Option

sparkyengine
authored and
zsxwing
committed
[SPARK-12711][ML] ML StopWordsRemover does not protect itself from column name duplication

grzegorz-chilkiewicz
authored and
jkbradley
committed
[SPARK-13056][SQL] map column would throw NPE if value is null

adrian-wang
authored and
marmbrus
committed
[DOCS] Update StructType.scala

swkimme
authored and
marmbrus
committed

Commits on Feb 3, 2016

[SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Adam Budde
authored and
Andrew Or
committed
[SPARK-12739][STREAMING] Details of batch in Streaming tab uses two Duration columns

mariobriggs
authored and
zsxwing
committed

Commits on Feb 4, 2016

Commits on Feb 5, 2016

[SPARK-13214][DOCS] update dynamicAllocation documentation

Bill Chambers
authored and
Andrew Or
committed

Commits on Feb 8, 2016

[SPARK-13210][SQL] catch OOM when allocate memory and expand array

Davies Liu
authored and
JoshRosen
committed

Commits on Feb 9, 2016

[SPARK-12807][YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

steveloughran
authored and
Marcelo Vanzin
committed

Commits on Feb 10, 2016

Commits on Feb 11, 2016

[SPARK-13274] Fix Aggregator Links on GroupedDataset Scala API

raelawang
authored and
rxin
committed
[SPARK-13265][ML] Refactoring of basic ML import/export for other file system besides HDFS

yu-iskw
authored and
mengxr
committed

Commits on Feb 12, 2016

[SPARK-13047][PYSPARK][ML] Pyspark Params.hasParam should not throw an error

sethah
authored and
mengxr
committed
[SPARK-13153][PYSPARK] ML persistence failed when handle no default value parameter

Wenpei
authored and
mengxr
committed

Commits on Feb 13, 2016

[SPARK-13142][WEB UI] Problem accessing Web UI /logPage/ on Microsoft Windows

markpavey
authored and
srowen
committed
[SPARK-12363][MLLIB] Remove setRun and fix PowerIterationClustering failed test

viirya
authored and
mengxr
committed

Commits on Feb 14, 2016

[SPARK-13300][DOCUMENTATION] Added pygments.rb dependancy

amitdev
authored and
srowen
committed

Commits on Feb 15, 2016

[SPARK-13312][MLLIB] Update java train-validation-split example in ml-guide

JeremyNixon
authored and
srowen
committed

Commits on Feb 16, 2016

Correct SparseVector.parse documentation

mgyucht
authored and
srowen
committed

Commits on Feb 17, 2016

[SPARK-13279] Remove O(n^2) operation from scheduler.

Sital Kedia
authored and
kayousterhout
committed
[SPARK-13350][DOCS] Config doc updated to state that PYSPARK_PYTHON's default is "python2.7"

chrisaycock
authored and
JoshRosen
committed

Commits on Feb 18, 2016

[SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

srowen
authored and
Andrew Or
committed

Commits on Feb 22, 2016

[SPARK-12546][SQL] Change default number of open parquet files
marmbrus
committed

Commits on Feb 23, 2016

Commits on Feb 24, 2016

Commits on Feb 25, 2016

[SPARK-13482][MINOR][CONFIGURATION] Make consistency of the configuraiton named in TransportConf.

SaintBacchus
authored and
rxin
committed
[SPARK-13473][SQL] Don't push predicate through project with nondeterministic field(s)

liancheng
authored and
cloud-fan
committed
[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames

Oliver Pierson
authored and
srowen
committed
[SPARK-13441][YARN] Fix NPE in yarn Client.createConfArchive method

chtyim
authored and
srowen
committed
[SPARK-13439][MESOS] Document that spark.mesos.uris is comma-separated

Michael Gummelt
authored and
srowen
committed
[SPARK-12316] Wait a minutes to avoid cycle calling.

SaintBacchus
authored and
Tom Graves
committed
Revert "[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames"
mengxr
committed
[SPARK-12874][ML] ML StringIndexer does not protect itself from column name duplication

yu-iskw
authored and
mengxr
committed

Commits on Feb 26, 2016

[SPARK-13454][SQL] Allow users to drop a table with a name starting with an underscore.
yhuai
committed

Commits on Feb 29, 2016

[SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype

thomastechs
authored and
yhuai
committed

Commits on Mar 3, 2016

[SPARK-13465] Add a task failure listener to TaskContext

Davies Liu
authored and
davies
committed
[SPARK-13601] call failure callbacks before writer.close()
davies
committed

Commits on Mar 4, 2016

[SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions

Davies Liu
authored and
davies
committed
[SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled
zsxwing
committed
[SPARK-11515][ML] QuantileDiscretizer should take random seed

yu-iskw
authored and
srowen
committed
[SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype mapping

thomastechs
authored and
yhuai
committed
[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames

Oliver Pierson
authored and
mengxr
committed
[HOTFIX] fix the conflict when cherry-pick
davies
committed

Commits on Mar 6, 2016

[SPARK-13697] [PYSPARK] Fix the missing module name of TransformFunctionSerializer.loads

zsxwing
authored and
davies
committed

Commits on Mar 7, 2016

[SPARK-13705][DOCS] UpdateStateByKey Operation documentation incorrectly refers to StatefulNetworkWordCount

rmishra
authored and
srowen
committed
[SPARK-13599][BUILD] remove transitive groovy dependencies from spark-hive and spark-hiveserver (branch 1.6)

steveloughran
authored and
srowen
committed
[MINOR][DOC] improve the doc for "spark.memory.offHeap.size"

CodingCat
authored and
zsxwing
committed
[SPARK-13648] Add Hive Cli to classes for isolated classloader

preecet
authored and
marmbrus
committed

Commits on Mar 8, 2016

[SPARK-13711][CORE] Don't call SparkUncaughtExceptionHandler in AppClient as it's in driver
zsxwing
committed

Commits on Mar 9, 2016

[SPARK-13755] Escape quotes in SQL plan visualization node labels
JoshRosen
committed
[SPARK-13631][CORE] Thread-safe getLocationsWithLargestOutputs

Andy Sloane
authored and
srowen
committed
[SPARK-13242] [SQL] codegen fallback in case-when if there many branches

Davies Liu
authored and
davies
committed

Commits on Mar 10, 2016

Commits on Mar 11, 2016

[MINOR][DOC] Fix supported hive version in doc

dongjoon-hyun
authored and
rxin
committed
[SPARK-13327][SPARKR] Added parameter validations for colnames<-

Oscar D. Lara Yejas
authored and
shivaram
committed

Commits on Mar 13, 2016

[SPARK-13810][CORE] Add Port Configuration Suggestions on Bind Exceptions

bjornjon
authored and
srowen
committed

Commits on Mar 14, 2016

[SQL] fix typo in DataSourceRegister

Jacky Li
authored and
rxin
committed
[MINOR][DOCS] Added Missing back slashes

danielsan
authored and
Andrew Or
committed
[MINOR][COMMON] Fix copy-paste oversight in variable naming

bjornjon
authored and
Andrew Or
committed

Commits on Mar 15, 2016

[SPARK-13207][SQL][BRANCH-1.6] Make partitioning discovery ignore _SUCCESS files.

yhuai
authored and
rxin
committed
[SPARK-13803] restore the changes in SPARK-3411

CodingCat
authored and
srowen
committed

Commits on Mar 17, 2016

[SPARK-13901][CORE] correct the logDebug information when jump to the next locality level

trueyao
authored and
srowen
committed

Commits on Mar 18, 2016

[SPARK-13958] Executor OOM due to unbounded growth of pointer array in…

Sital Kedia
authored and
davies
committed

Commits on Mar 22, 2016

[SPARK-14058][PYTHON] Incorrect docstring in Window.order

zero323
authored and
rxin
committed
[SPARK-13772][SQL] Fix data type mismatch for decimal

cenyuhai
authored and
liancheng
committed
[SPARK-14006][SPARKR] Fix SparkR lint-r test errors in branch-1.6.

Sun Rui
authored and
mengxr
committed
[SPARK-13806] [SQL] fix rounding mode of negative float/double

Davies Liu
authored and
davies
committed

Commits on Mar 23, 2016

[SPARK-13642][YARN][1.6-BACKPORT] Properly handle signal kill in ApplicationMaster

jerryshao
authored and
Tom Graves
committed
[SPARK-14074][SPARKR] Specify commit sha1 ID when using install_github to install intr package.

Sun Rui
authored and
mengxr
committed

Commits on Mar 25, 2016

Commits on Mar 28, 2016

[SPARK-14187][MLLIB] Fix incorrect use of binarySearch in SparseMatrix

Chenliang Xu
authored and
mengxr
committed
[SPARK-13622][YARN] Issue creating level db for YARN shuffle service

ashangit
authored and
Marcelo Vanzin
committed

Commits on Mar 29, 2016

[SPARK-14219][GRAPHX] Fix `pickRandomVertex` not to fall into infinit…

dongjoon-hyun
authored and
rxin
committed
[SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

jeanlyn
authored and
Andrew Or
committed
[SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

carsonwang
authored and
Andrew Or
committed

Commits on Mar 30, 2016

[SPARK-11507][MLLIB] add compact in Matrices fromBreeze

hhbyyh
authored and
jkbradley
committed

Commits on Mar 31, 2016

[SPARK-14138] [SQL] Fix generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

kiszk
authored and
davies
committed

Commits on Apr 4, 2016

[SPARK-11327][MESOS] Backport dispatcher does not respect all args f…

jayv
authored and
Andrew Or
committed

Commits on Apr 5, 2016

[SPARK-14368][PYSPARK] Support python.spark.worker.memory with upper-case unit.

yongtang
authored and
sarutak
committed
[SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks

jeanlyn
authored and
Andrew Or
committed

Commits on Apr 6, 2016

[SPARK-14322][MLLIB] Use treeAggregate instead of reduce in OnlineLDAOptimizer

hhbyyh
authored and
jkbradley
committed

Commits on Apr 8, 2016

[DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode.

Michael Gummelt
authored and
Andrew Or
committed
[SPARK-14468] Always enable OutputCommitCoordinator
Andrew Or
committed

Commits on Apr 10, 2016

[SPARK-14357][CORE] Properly handle the root cause being a commit denied exception

jasonmoore2k
authored and
Andrew Or
committed

Commits on Apr 12, 2016

[SPARK-14242][CORE][NETWORK] avoid copy in compositeBuffer for frame decoder

liyezhang556520
authored and
davies
committed
[SPARK-14563][ML] use a random table name instead of __THIS__ in SQLTransformer
mengxr
committed
[SPARK-14544] [SQL] improve performance of SQL UI tab

Davies Liu
authored and
davies
committed
[SPARK-14363] Fix executor OOM due to memory leak in the Sorter

Sital Kedia
authored and
davies
committed

Commits on Apr 14, 2016

[SPARK-14618][ML][DOC] Updated RegressionEvaluator.metricName param doc
jkbradley
committed

Commits on Apr 15, 2016

[SPARK-14665][ML][PYTHON] Fixed bug with StopWordsRemover default stopwords
jkbradley
committed

Commits on Apr 19, 2016

[SPARK-13227] Risky apply() in OpenHashMap

CodingCat
authored and
rxin
committed

Commits on Apr 20, 2016

[SPARK-14679][UI] Fix UI DAG visualization OOM.

rdblue
authored and
srowen
committed

Commits on Apr 21, 2016

[SPARK-14739][PYSPARK] Fix Vectors parser bugs

Arash Parsa
authored and
srowen
committed
[SPARK-14787][SQL] Upgrade Joda-Time library from 2.9 to 2.9.3

HyukjinKwon
authored and
srowen
committed

Commits on Apr 26, 2016

[SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula - 1.6 backport
jkbradley
committed

Commits on Apr 27, 2016

Commits on Apr 28, 2016

[SPARK-14965][SQL] Indicate an exception is thrown for a missing struct field

Gregory Hart
authored and
rxin
committed

Commits on Apr 30, 2016

[SPARK-14757] [SQL] Fix nullability bug in EqualNullSafe codegen

rxin
authored and
davies
committed

Commits on May 3, 2016

[SPARK-14897][CORE] Upgrade Jetty to latest version of 8
srowen
committed

Commits on May 5, 2016

[SPARK-14915][CORE] Don't re-queue a task if another attempt has already succeeded

jasonmoore2k
authored and
srowen
committed

Commits on May 6, 2016

Commits on May 9, 2016

[SPARK-15223][DOCS] fix wrongly named config reference

philipphoffmann
authored and
Andrew Or
committed
[SPARK-15209] Fix display of job descriptions with single quotes in web UI timeline

JoshRosen
authored and
sarutak
committed

Commits on May 11, 2016

[SPARK-14495][SQL][1.6] fix resolution failure of having clause with distinct aggregate function

xwu0226
authored and
cloud-fan
committed
[SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state

zsxwing
authored and
Andrew Or
committed
[SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times

zsxwing
authored and
Andrew Or
committed
[SPARK-13522][CORE] Fix the exit log place for heartbeat

zsxwing
authored and
Andrew Or
committed
[SPARK-15262] Synchronize block manager / scheduler executor state

Andrew Or
authored and
zsxwing
committed

Commits on May 12, 2016

[SPARK-15260] Atomically resize memory pools (branch 1.6)

Andrew Or
authored and
davies
committed

Commits on May 20, 2016

Commits on May 26, 2016

[SPARK-10722] RDDBlockId not found in driver-heartbeater

simonjscott
authored and
srowen
committed
[SPARK-8428][SPARK-13850] Fix integer overflows in TimSort

sameeragarwal
authored and
rxin
committed

Commits on May 27, 2016

[BUILD][1.6] Fix compilation

sameeragarwal
authored and
rxin
committed

Commits on May 31, 2016

[SPARK-15528][SQL] Fix race condition in NumberConverter

maropu
authored and
srowen
committed

Commits on Jun 1, 2016

[SPARK-15601][CORE] CircularBuffer's toString() to print only the contents written if buffer isn't full

tejasapatil
authored and
srowen
committed

Commits on Jun 2, 2016

[SPARK-14204][SQL] register driverClass rather than user-specified class

mchalek
authored and
srowen
committed

Commits on Jun 3, 2016

[SPARK-15736][CORE][BRANCH-1.6] Gracefully handle loss of DiskStore files

JoshRosen
authored and
Andrew Or
committed
[SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential.

Subroto Sanyal
authored and
Marcelo Vanzin
committed

Commits on Jun 5, 2016

[SPARK-15723] Fixed local-timezone-brittle test where short-timezone form "EST" is …

javabrett
authored and
srowen
committed

Commits on Jun 7, 2016

[SPARK-12655][GRAPHX] GraphX does not unpersist RDDs

jasoncl
authored and
srowen
committed

Commits on Jun 9, 2016

Commits on Jun 10, 2016

[DOCUMENTATION] fixed groupby aggregation example for pyspark

mortada
authored and
rxin
committed

Commits on Jun 12, 2016

[SPARK-15892][ML] Incorrectly merged AFTAggregator with zero total count

HyukjinKwon
authored and
jkbradley
committed

Commits on Jun 14, 2016

Revert "[SPARK-15892][ML] Incorrectly merged AFTAggregator with zero total count"
jkbradley
committed

Commits on Jun 15, 2016

[SPARK-15915][SQL] Logical plans should use subqueries eliminated plan when override sameResult.

ueshin
authored and
cloud-fan
committed

Commits on Jun 17, 2016

Revert "[SPARK-15395][CORE] Use getHostString to create RpcAddress (backport for 1.6)"
zsxwing
committed

Commits on Jun 18, 2016

[SPARK-15892][ML] Backport correctly merging AFTAggregators to branch 1.6

HyukjinKwon
authored and
mengxr
committed
[SPARK-16035][PYSPARK] Fix SparseVector parser assertion for end parenthesis

andreapasqua
authored and
mengxr
committed

Commits on Jun 19, 2016

Commits on Jun 20, 2016

[SPARK-14391][LAUNCHER] Fix launcher communication test, take 2.
Marcelo Vanzin
committed
[SPARK-15613] [SQL] Fix incorrect days to millis conversion due to Daylight Saving Time

Davies Liu
authored and
davies
committed

Commits on Jun 21, 2016

Commits on Jun 22, 2016

[SPARK-6005][TESTS] Fix flaky test: o.a.s.streaming.kafka.DirectKafkaStreamSuite.offset recovery

zsxwing
authored and
srowen
committed

Commits on Jun 24, 2016

[SPARK-16077] [PYSPARK] catch the exception from pickle.whichmodule()

Davies Liu
authored and
davies
committed

Commits on Jun 25, 2016

Commits on Jun 27, 2016

[SPARK-16214][EXAMPLES] fix the denominator of SparkPi

yanghaogn
authored and
srowen
committed

Commits on Jun 28, 2016

Commits on Jun 29, 2016

[SPARK-16044][SQL] Backport input_file_name() for data source based on NewHadoopRDD to branch 1.6

HyukjinKwon
authored and
rxin
committed

Commits on Jun 30, 2016

[SPARK-16257][BUILD] Update spark_ec2.py to support Spark 1.6.2 and 1.6.3.

Brian Uri
authored and
srowen
committed

Commits on Jul 1, 2016

Commits on Jul 4, 2016

[SPARK-16329][SQL][BACKPORT-1.6] Star Expansion over Table Containing No Column #14040

gatorsmile
authored and
cloud-fan
committed
[SPARK-16353][BUILD][DOC] Missing javadoc options for java unidoc

Michael Allman
authored and
srowen
committed

Commits on Jul 6, 2016

Commits on Jul 7, 2016

Commits on Jul 11, 2016

[SPARK-16385][CORE] Catch correct exception when calling method via reflection.

Marcelo Vanzin
authored and
srowen
committed

Commits on Jul 13, 2016

Commits on Jul 14, 2016

[SPARK-16313][SQL][BRANCH-1.6] Spark should not silently drop exceptions in file listing
yhuai
committed

Commits on Jul 20, 2016

[SPARK-16440][MLLIB] Destroy broadcasted variables even on driver

Anthony Truchet
authored and
srowen
committed