Resync apache#253

Merged

robert3005 merged 189 commits intomasterfrom

Sep 15, 2017

Commits on Aug 17, 2017

Commits on Aug 18, 2017

[SPARK-21739][SQL] Cast expression should initialize timezoneId when it is called statically to convert something into TimestampType

DonnyZone
authored and
gatorsmile
committed
[SPARK-21778][SQL] Simpler Dataset.sample API in Scala / Java

rxin
authored and
HyukjinKwon
committed
[SPARK-21213][SQL] Support collecting partition-level statistics: rowCount and sizeInBytes

mbasmanova
authored and
gatorsmile
committed
[SPARK-21743][SQL][FOLLOW-UP] top-most limit should not cause memory leak

cloud-fan
authored and
gatorsmile
committed
[MINOR][TYPO] Fix typos: runnning and Excecutors

ash211
authored and
rxin
committed

Commits on Aug 19, 2017

[SPARK-21566][SQL][PYTHON] Python method for summary

aray
authored and
holdenk
committed
[SPARK-21790][TESTS] Fix Docker-based Integration Test errors.

wangyum
authored and
gatorsmile
committed

Commits on Aug 20, 2017

Commits on Aug 21, 2017

[SPARK-21782][CORE] Repartition creates skews when numPartitions is a power of 2

megaserg
authored and
srowen
committed
[SPARK-21718][SQL] Heavy log of type: "Skipping partition based on stats ..."

srowen
authored and
hvanhovell
committed
[SPARK-21468][PYSPARK][ML] Python API for FeatureHasher
Nick Pentreath
committed
[SPARK-21790][TESTS][FOLLOW-UP] Add filter pushdown verification back.

wangyum
authored and
gatorsmile
committed
[SPARK-21617][SQL] Store correct table metadata when altering schema in Hive metastore.

Marcelo Vanzin
authored and
gatorsmile
committed

Commits on Aug 22, 2017

[SPARK-19762][ML][FOLLOWUP] Add necessary comments to L2Regularization.
yanboliang
committed
[SPARK-21070][PYSPARK] Attempt to update cloudpickle again

rgbkrk
authored and
HyukjinKwon
committed
[SPARK-21584][SQL][SPARKR] Update R method for summary to call new implementation

aray
authored and
Felix Cheung
committed
[SPARK-21803][TEST] Remove the HiveDDLCommandSuite

gatorsmile
authored and
cloud-fan
committed
[SPARK-20641][CORE] Add missing kvstore module in Laucher and SparkSubmit code

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-21499][SQL] Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)
gatorsmile
committed
[SPARK-21769][SQL] Add a table-specific option for always respecting schemas inferred/controlled by Spark SQL

gatorsmile
authored and
sameeragarwal
committed
[SPARK-21681][ML] fix bug of MLOR do not work correctly when featureStd contains zero

WeichenXu123
authored and
jkbradley
committed

Commits on Aug 23, 2017

[SPARK-10931][ML][PYSPARK] PySpark Models Copy Param Values from Estimator

BryanCutler
authored and
holdenk
committed
[SPARK-21765] Set isStreaming on leaf nodes for streaming plans.

Jose Torres
authored and
tdas
committed
[ML][MINOR] Make sharedParams update.
yanboliang
committed
[SPARK-19326] Speculated task attempts do not get launched in few scenarios

janewangfb
authored and
cloud-fan
committed
[SPARK-12664][ML] Expose probability in mlp model

WeichenXu123
authored and
jkbradley
committed
[SPARK-21501] Change CacheLoader to limit entries based on memory footprint

Sanket Chintapalli
authored and
Tom Graves
committed
[SPARK-21603][SQL][FOLLOW-UP] Change the default value of maxLinesPerFunction into 4000

maropu
authored and
gatorsmile
committed

Commits on Aug 24, 2017

[SPARK-21807][SQL] Override ++ operation in ExpressionSet to reduce clone time

eatoncys
authored and
gatorsmile
committed
[SPARK-21805][SPARKR] Disable R vignettes code on Windows

felixcheung
authored and
Felix Cheung
committed
[SPARK-21694][MESOS] Support Mesos CNI network labels

susanxhuynh
authored and
srowen
committed
[MINOR][SQL] The comment of Class ExchangeCoordinator exist a typing and context error

figo77
authored and
srowen
committed
[SPARK-21804][SQL] json_tuple returns null values within repeated columns except the first one

jmchung
authored and
HyukjinKwon
committed
[SPARK-19165][PYTHON][SQL] PySpark APIs using columns as arguments should validate input types for column
HyukjinKwon
committed
[SPARK-21745][SQL] Refactor ColumnVector hierarchy to make ColumnVector read-only and to introduce WritableColumnVector.

ueshin
authored and
cloud-fan
committed
[SPARK-21759][SQL] In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

viirya
authored and
cloud-fan
committed
[SPARK-21826][SQL] outer broadcast hash join should not throw NPE

cloud-fan
authored and
hvanhovell
committed
[SPARK-21788][SS] Handle more exceptions when stopping a streaming query

zsxwing
authored and
tdas
committed
[SPARK-21701][CORE] Enable RPC client to use ` SO_RCVBUF` and ` SO_SNDBUF` in SparkConf.

xu.zhang
authored and
zsxwing
committed
[SPARK-21830][SQL] Bump ANTLR version and fix a few issues.

hvanhovell
authored and
gatorsmile
committed

Commits on Aug 25, 2017

[SPARK-21108][ML] convert LinearSVC to aggregator framework

YY-OnCall
authored and
yanboliang
committed
[SPARK-21255][SQL][WIP] Fixed NPE when creating encoder for enum

mike0sv
authored and
srowen
committed
[SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache

caneGuy
authored and
cloud-fan
committed
[MINOR][BUILD] Fix build warnings and Java lint errors
srowen
committed
[SPARK-21832][TEST] Merge SQLBuilderTest into ExpressionSQLBuilderSuite

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-17742][CORE] Fail launcher app handle if child process exits with error.
Marcelo Vanzin
committed
[SPARK-21756][SQL] Add JSON option to allow unquoted control characters

vinodkc
authored and
gatorsmile
committed
[SPARK-21837][SQL][TESTS] UserDefinedTypeSuite Local UDTs not actually testing what it intends

srowen
authored and
gatorsmile
committed

Commits on Aug 26, 2017

Commits on Aug 27, 2017

[SPARK-21843] testNameNote should be "(minNumPostShufflePartitions: 5)"

iamhumanbeing
authored and
srowen
committed

Commits on Aug 28, 2017

[SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSummarizer.variance generate negative result

WeichenXu123
authored and
srowen
committed
[SPARK-21798] No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

pgandhi
authored and
Tom Graves
committed
[SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler Unit Test coverage for different build cases

erenavsarogullari
authored and
squito
committed
[SPARK-17139][ML] Add model summary for MultinomialLogisticRegression

WeichenXu123
authored and
jkbradley
committed

Commits on Aug 29, 2017

[SPARK-21781][SQL] Modify DataSourceScanExec to use concrete ColumnVector type.

ueshin
authored and
cloud-fan
committed
[SPARK-21848][SQL] Add trait UserDefinedExpression to identify user-defined functions

gengliangwang
authored and
gatorsmile
committed
[SPARK-21255][SQL] simplify encoder for java enum

cloud-fan
authored and
gatorsmile
committed
[SPARK-21801][SPARKR][TEST] unit test randomly fail with randomforest

felixcheung
authored and
Felix Cheung
committed
[MINOR][ML] Document treatment of instance weights in logreg summary
jkbradley
committed
[SPARK-21728][CORE] Allow SparkSubmit to use Logging.
Marcelo Vanzin
committed
[SPARK-21813][CORE] Modify TaskMemoryManager.MAXIMUM_PAGE_SIZE_BYTES comments

Geek-He
authored and
srowen
committed

Commits on Aug 30, 2017

[SPARK-21845][SQL] Make codegen fallback of expressions configurable
gatorsmile
committed
[SPARK-20886][CORE] HadoopMapReduceCommitProtocol to handle FileOutputCommitter.getWorkPath==null

steveloughran
authored and
HyukjinKwon
committed
Merge remote-tracking branch 'origin/master' into palantir-master
ash211
committed
Resolve conflicts
ash211
committed
Fix broken Kubernetes code
ash211
committed
[MINOR][TEST] Off -heap memory leaks for unit tests

10110346
authored and
srowen
committed
[SPARK-21873][SS] - Avoid using `return` inside `CachedKafkaConsumer.get`

Yuval Itzchakov
authored and
srowen
committed
[SPARK-21806][MLLIB] BinaryClassificationMetrics pr(): first point (0.0, 1.0) is misleading
srowen
committed
[SPARK-21764][TESTS] Fix tests failures on Windows: resources not being closed and incorrect paths
HyukjinKwon
committed
[SPARK-21469][ML][EXAMPLES] Adding Examples for FeatureHasher

BryanCutler
authored and
Nick Pentreath
committed
Revert "[SPARK-21845][SQL] Make codegen fallback of expressions configurable"
gatorsmile
committed
[MINOR][SQL][TEST] Test shuffle hash join while is not expected

heary-cao
authored and
gatorsmile
committed
Fix Java style bugs
ash211
committed
[SPARK-21834] Incorrect executor request in case of dynamic allocation

Sital Kedia
authored and
Marcelo Vanzin
committed
[SPARK-21839][SQL] Support SQL config for ORC compression

dongjoon-hyun
authored and
HyukjinKwon
committed

Commits on Aug 31, 2017

[SPARK-21875][BUILD] Fix Java style bugs

ash211
authored and
HyukjinKwon
committed
[SPARK-11574][CORE] Add metrics StatsD sink

Xiaofeng Lin
authored and
jerryshao
committed
Our newer parquet is 40bytes more efficient: 1.9.1-palantir3 vs 1.8.2
ash211
committed
[SPARK-17321][YARN] Avoid writing shuffle metadata to disk if NM recovery is disabled
jerryshao
committed
[SPARK-21534][SQL][PYSPARK] PickleException when creating dataframe from python row with empty bytearray

viirya
authored and
HyukjinKwon
committed
[SPARK-21583][SQL] Create a ColumnarBatch from ArrowColumnVectors

BryanCutler
authored and
ueshin
committed
[SPARK-21878][SQL][TEST] Create SQLMetricsTestUtils
gatorsmile
committed
[SPARK-21886][SQL] Use SparkSession.internalCreateDataFrame to create…

jaceklaskowski
authored and
gatorsmile
committed
[SPARK-20812][MESOS] Add secrets support to the dispatcher

ArtRand
authored and
Marcelo Vanzin
committed
[SPARK-21583][HOTFIX] Removed intercept in test causing failures

BryanCutler
authored and
gatorsmile
committed
[SPARK-17107][SQL][FOLLOW-UP] Remove redundant pushdown rule for Union
gatorsmile
committed
[SPARK-21110][SQL] Structs, arrays, and other orderable datatypes should be usable in inequalities

aray
authored and
gatorsmile
committed
[SPARK-17139][ML][FOLLOW-UP] Add convenient method `asBinary` for casting to BinaryLogisticRegressionSummary

WeichenXu123
authored and
jkbradley
committed
[SPARK-21862][ML] Add overflow check in PCA

WeichenXu123
authored and
jkbradley
committed

Commits on Sep 1, 2017

Commits on Sep 2, 2017

Commits on Sep 3, 2017

[SPARK-21897][PYTHON][R] Add unionByName API to DataFrame in Python and R
HyukjinKwon
committed

Commits on Sep 4, 2017

Commits on Sep 5, 2017

Commits on Sep 6, 2017

[SPARK-18061][THRIFTSERVER] Add spnego auth support for ThriftServer thrift/http protocol
jerryshao
committed
[SPARK-9104][CORE] Expose Netty memory metrics in Spark

jerryshao
authored and
zsxwing
committed
[SPARK-21924][DOCS] Update structured streaming programming guide doc

Riccardo Corbella
authored and
srowen
committed
[SPARK-19357][ML] Adding parallel model evaluation in ML tuning

BryanCutler
authored and
Nick Pentreath
committed
[SPARK-21903][BUILD][FOLLOWUP] Upgrade scalastyle-maven-plugin and scalastyle as well in POM and SparkBuild.scala
HyukjinKwon
committed
[SPARK-21835][SQL] RewritePredicateSubquery should not produce unresolved query plans

viirya
authored and
gatorsmile
committed
[SPARK-21801][SPARKR][TEST] set random seed for predictable test

felixcheung
authored and
Felix Cheung
committed
[SPARK-21765] Check that optimization doesn't affect isStreaming bit.

jose-torres
authored and
tdas
committed
[SPARK-21901][SS] Define toString for StateOperatorProgress

jaceklaskowski
authored and
zsxwing
committed

Commits on Sep 7, 2017

Fixed pandoc dependency issue in python/setup.py

Tucker Beck
authored and
HyukjinKwon
committed
[SPARK-21835][SQL][FOLLOW-UP] RewritePredicateSubquery should not produce unresolved query plans

viirya
authored and
gatorsmile
committed
[SPARK-21912][SQL] ORC/Parquet table should not create invalid column names

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-21890] Credentials not being passed to add the tokens

Sanket Chintapalli
authored and
Tom Graves
committed
[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs

dongjoon-hyun
authored and
gatorsmile
committed

Commits on Sep 8, 2017

Commits on Sep 9, 2017

Commits on Sep 10, 2017

[SPARK-20098][PYSPARK] dataType's typeName fix

szalai1
authored and
HyukjinKwon
committed

Commits on Sep 11, 2017

Commits on Sep 12, 2017

[SPARK-21976][DOC] Fix wrong documentation for Mean Absolute Error.

FavioVazquez
authored and
srowen
committed
[SPARK-14516][ML] Adding ClusteringEvaluator with the implementation of Cosine silhouette and squared Euclidean silhouette.

mgaido91
authored and
yanboliang
committed
[SPARK-21610][SQL][FOLLOWUP] Corrupt records are not handled properly when creating a dataframe from a file

jmchung
authored and
HyukjinKwon
committed
[DOCS] Fix unreachable links in the document

sarutak
authored and
srowen
committed
[SPARK-17642][SQL] support DESC EXTENDED/FORMATTED table column commands

wzhfy
authored and
gatorsmile
committed
[SPARK-21027][ML][PYTHON] Added tunable parallelism to one vs. rest in both Scala mllib and Pyspark

ajaysaini725
authored and
jkbradley
committed
[SPARK-21368][SQL] TPCDSQueryBenchmark can't refer query files.

sarutak
authored and
gatorsmile
committed
[SPARK-18608][ML] Fix double caching

zhengruifeng
authored and
jkbradley
committed
[SPARK-21979][SQL] Improve QueryPlanConstraints framework

gengliangwang
authored and
gatorsmile
committed

Commits on Sep 13, 2017

[SPARK-21513][SQL] Allow UDF to_json support converting MapType to json

goldmedal
authored and
HyukjinKwon
committed
[SPARK-21027][MINOR][FOLLOW-UP] add missing since tag

WeichenXu123
authored and
srowen
committed
[BUILD] Close stale PRs
srowen
committed
[SPARK-21982] Set locale to US

Gschiavon
authored and
srowen
committed
[SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behind a profile
srowen
committed
[SPARK-21963][CORE][TEST] Create temp file should be delete after use

heary-cao
authored and
srowen
committed
[SPARK-21690][ML] one-pass imputer

zhengruifeng
authored and
yanboliang
committed
[SPARK-21970][CORE] Fix Redundant Throws Declarations in Java Codebase

original-brownbear
authored and
srowen
committed
[SPARK-21980][SQL] References in grouping functions should be indexed with semanticEquals

DonnyZone
authored and
gatorsmile
committed
[SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.scala

janewangfb
authored and
gatorsmile
committed
[SPARK-20427][SQL] Read JDBC table use custom schema

wangyum
authored and
gatorsmile
committed

Commits on Sep 14, 2017

[SPARK-21973][SQL] Add an new option to filter queries in TPC-DS

maropu
authored and
gatorsmile
committed
[MINOR][SQL] Only populate type metadata for required types such as CHAR/VARCHAR.

dilipbiswal
authored and
gatorsmile
committed
[SPARK-21854] Added LogisticRegressionTrainingSummary for MultinomialLogisticRegression in Python API

Ming Jiang
authored and
yanboliang
committed
[MINOR][DOC] Add missing call of `update()` in examples of PeriodicGraphCheckpointer & PeriodicRDDCheckpointer

zhengruifeng
authored and
yanboliang
committed
[SPARK-18608][ML][FOLLOWUP] Fix double caching for PySpark OneVsRest.
yanboliang
committed
[SPARK-4131][FOLLOW-UP] Support "Writing data into the filesystem from queries"

gatorsmile
authored and
cloud-fan
committed
[SPARK-21922] Fix duration always updating when task failed but status is still RUN…

caneGuy
authored and
jerryshao
committed
[SPARK-17642][SQL][FOLLOWUP] drop test tables and improve comments

Zhenhua Wang
authored and
cloud-fan
committed
[SPARK-21988] Add default stats to StreamingExecutionRelation.

jose-torres
authored and
zsxwing
committed

Commits on Sep 15, 2017

[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR

goldmedal
authored and
HyukjinKwon
committed
[SPARK-22018][SQL] Preserve top-level alias metadata when collapsing projects
tdas
committed
[SPARK-21902][CORE] Print root cause for BlockManager#doPut

caneGuy
authored and
jerryshao
committed
[SPARK-22002][SQL] Read JDBC table use custom schema support specify partial fields.

wangyum
authored and
gatorsmile
committed
[SPARK-21987][SQL] fix a compatibility issue of sql event logs

cloud-fan
authored and
gatorsmile
committed
Merge branch 'master' into resync-apache
Robert Kruszewski
committed
Resolve conflicts
Robert Kruszewski
committed
manifset
Robert Kruszewski
committed
checkstyle
Robert Kruszewski
committed
order
Robert Kruszewski
committed
correctly resolve conflicts
Robert Kruszewski
committed