Update from original#1

Merged

jayadevanmurali merged 313 commits intojayadevanmurali:masterfrom

Jan 31, 2016

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 8, 2016

fixed numVertices in transitive closure example

udoklein
authored and
srowen
committed
[SPARK-12654] sc.wholeTextFiles with spark.hadoop.cloneConf=true fail…

tgravescs
authored and
Tom Graves
committed
[SPARK-4819] Remove Guava's "Optional" from public API

srowen
authored and
rxin
committed
[SPARK-12593][SQL] Converts resolved logical plan back to SQL

liancheng
authored and
yhuai
committed

Commits on Jan 9, 2016

Commits on Jan 11, 2016

[SPARK-12734][BUILD] Fix Netty exclusion and use Maven Enforcer to prevent future bugs
JoshRosen
committed
[SPARK-3873][BUILD] Enable import ordering error checking.

Marcelo Vanzin
authored and
rxin
committed
[SPARK-4628][BUILD] Add a resolver to MiMaBuild.scala for mqttv3(1.0.1).

sarutak
authored and
rxin
committed
[SPARK-12734][HOTFIX][TEST-MAVEN] Fix bug in Netty exclusions
JoshRosen
committed
[SPARK-12539][FOLLOW-UP] always sort in partitioning writer

cloud-fan
authored and
rxin
committed
removed lambda from sortByKey()

udoklein
authored and
srowen
committed
[SPARK-12269][STREAMING][KINESIS] Update aws-java-sdk version

BrianLondon
authored and
srowen
committed
[SPARK-12744][SQL] Change parsing JSON integers to timestamps to treat integers as number of seconds

antlypls
authored and
yhuai
committed
[STREAMING][MINOR] Typo fixes

jaceklaskowski
authored and
zsxwing
committed
[SPARK-12734][HOTFIX] Build changes must trigger all tests; clean after install in dep tests
JoshRosen
committed
[SPARK-12758][SQL] add note to Spark SQL Migration guide about TimestampType casting

blbradley
authored and
marmbrus
committed
[SPARK-12603][MLLIB] PySpark MLlib GaussianMixtureModel should support single instance predict/predictSoft

yanboliang
authored and
jkbradley
committed
[SPARK-12685][MLLIB] word2vec trainWordsCount gets overflow

hhbyyh
authored and
jkbradley
committed
[SPARK-10809][MLLIB] Single-document topicDistributions method for LocalLDAModel

hhbyyh
authored and
jkbradley
committed

Commits on Jan 12, 2016

Commits on Jan 13, 2016

[SPARK-12785][SQL] Add ColumnarBatch, an in memory columnar format for execution.

nongli
authored and
rxin
committed
[SPARK-12788][SQL] Simplify BooleanEquality by using casts.
rxin
committed
[SPARK-12692][BUILD][CORE] Scala style: Fix the style violation (Space before ",")

sarutak
authored and
rxin
committed
[SPARK-12558][SQL] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12692][BUILD][SQL] Scala style: Fix the style violation (Space before ",")

sarutak
authored and
rxin
committed
[SPARK-12692][BUILD] Enforce style checking about white space before comma

sarutak
authored and
rxin
committed
[SPARK-12692][BUILD][HOT-FIX] Fix the scala style of KinesisBackedBlockRDDSuite.scala.
yhuai
committed
[SPARK-9297] [SQL] Add covar_pop and covar_samp

viirya
authored and
davies
committed
[SPARK-12805][MESOS] Fixes documentation on Mesos run modes

Luc Bourlier
authored and
rxin
committed
[SPARK-12761][CORE] Remove duplicated code

jodersky
authored and
rxin
committed
[SPARK-9383][PROJECT-INFRA] PR merge script should reset back to previous branch when possible
JoshRosen
committed
[SPARK-12268][PYSPARK] Make pyspark shell pythonstartup work under python3

erikselin
authored and
JoshRosen
committed
[SPARK-12642][SQL] improve the hash expression to be decoupled from unsafe row

cloud-fan
authored and
rxin
committed
[SPARK-12791][SQL] Simplify CaseWhen by breaking "branches" into "conditions" and "values"
rxin
committed
[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

carsonwang
authored and
JoshRosen
committed

Commits on Jan 14, 2016

Commits on Jan 15, 2016

[SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test

JoshRosen
authored and
Andrew Or
committed
[SPARK-12813][SQL] Eliminate serialization for back to back operations
marmbrus
committed
[SPARK-12708][UI] Sorting task error in Stages Page when yarn mode.

yoshidakuy
authored and
sarutak
committed
[SPARK-12829] Turn Java style checker on
rxin
committed
[SPARK-12830] Java style: disallow trailing whitespaces.
rxin
committed
[SPARK-12655][GRAPHX] GraphX does not unpersist RDDs

jasoncl
authored and
srowen
committed
[SPARK-2930] clarify docs on using webhdfs with spark.yarn.access.nam…

Tom Graves
authored and
srowen
committed
[SPARK-11031][SPARKR] Method str() on a DataFrame

Oscar D. Lara Yejas
authored and
shivaram
committed
[MINOR] [SQL] GeneratedExpressionCode -> ExprCode

Davies Liu
authored and
davies
committed
[SPARK-12833][SQL] Initial import of spark-csv

falaki
authored and
rxin
committed
[SPARK-12667] Remove block manager's internal "external block store" API

rxin
authored and
JoshRosen
committed
[SPARK-12833][HOT-FIX] Fix scala 2.11 compilation.
yhuai
committed
Fix typo

julienbaley
authored and
rxin
committed
[SPARK-12716][WEB UI] Add a TOTALS row to the Executors Web UI

ajbozarth
authored and
Tom Graves
committed
[SQL][MINOR] BoundReference do not need to be NamedExpression

cloud-fan
authored and
marmbrus
committed
[SPARK-12575][SQL] Grammar parity with existing SQL parser

hvanhovell
authored and
rxin
committed
[SPARK-11925][ML][PYSPARK] Add PySpark missing methods for ml.feature during Spark 1.6 QA

yanboliang
authored and
jkbradley
committed

Commits on Jan 16, 2016

[SPARK-12833][HOT-FIX] Reset the locale after we set it.
yhuai
committed
[SPARK-12842][TEST-HADOOP2.7] Add Hadoop 2.7 build profile

JoshRosen
authored and
rxin
committed
[SPARK-12649][SQL] support reading bucketed table

cloud-fan
authored and
rxin
committed
[SPARK-12644][SQL] Update parquet reader to be vectorized.

nongli
authored and
rxin
committed
[SPARK-12840] [SQL] Support passing arbitrary objects (not just expressions) into code generated classes

Davies Liu
authored and
davies
committed
[SPARK-12856] [SQL] speed up hashCode of unsafe array

cloud-fan
authored and
davies
committed
[SPARK-12722][DOCS] Fixed typo in Pipeline example

Jeff Lam
authored and
srowen
committed
[SPARK-12796] [SQL] Whole stage codegen

Davies Liu
authored and
davies
committed

Commits on Jan 17, 2016

[SPARK-12860] [SQL] speed up safe projection for primitive types

cloud-fan
authored and
davies
committed
[SPARK-12862][SPARKR] Jenkins does not run R tests

felixcheung
authored and
shivaram
committed
[SQL] [MINOR] speed up hashcode for UTF8String

cloud-fan
authored and
davies
committed

Commits on Jan 18, 2016

[SPARK-10264][DOCUMENTATION] Added @Since to ml.recomendation

Wenpei
authored and
srowen
committed
[SPARK-12558][FOLLOW-UP] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12873][SQL] Add more comment in HiveTypeCoercion for type widening
rxin
committed
[SPARK-12346][ML] Missing attribute names in GLM for vector-type features

ericl
authored and
mengxr
committed
[SPARK-12884] Move classes to their own files for readability

Andrew Or
authored and
rxin
committed
[SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager

JoshRosen
authored and
Andrew Or
committed
[SPARK-12855][SQL] Remove parser dialect developer API
rxin
committed
[SPARK-12841][SQL] fix cast in filter

cloud-fan
authored and
yhuai
committed
[SPARK-12882][SQL] simplify bucket tests and add more comments

cloud-fan
authored and
rxin
committed
[SPARK-12814][DOCUMENT] Add deploy instructions for Python in flume integration doc

zsxwing
authored and
tdas
committed

Commits on Jan 19, 2016

Revert "[SPARK-12829] Turn Java style checker on"
zsxwing
committed
[SPARK-12894][DOCUMENT] Add deploy instructions for Python in Kinesis integration doc

zsxwing
authored and
tdas
committed
[SPARK-12889][SQL] Rename ParserDialect -> ParserInterface.
rxin
committed
[SPARK-12700] [SQL] embed condition into SMJ and BroadcastHashJoin

Davies Liu
authored and
davies
committed
[SPARK-12885][MINOR] Rename 3 fields in ShuffleWriteMetrics

Andrew Or
authored and
JoshRosen
committed
[HOT][BUILD] Changed the import order

gatorsmile
authored and
JoshRosen
committed
[SPARK-12668][SQL] Providing aliases for CSV options to be similar to Pandas and R

HyukjinKwon
authored and
rxin
committed
[SQL][MINOR] Fix one little mismatched comment according to the codes in interface.scala

lw-lin
authored and
rxin
committed
[SPARK-7683][PYSPARK] Confusing behavior of fold function of RDD in pyspark
srowen
committed
[MLLIB] Fix CholeskyDecomposition assertion's message

wjur
authored and
srowen
committed
[SPARK-11944][PYSPARK][MLLIB] python mllib.clustering.bisecting k means

holdenk
authored and
jkbradley
committed
[SPARK-12870][SQL] better format bucket id in file name

cloud-fan
authored and
yhuai
committed
[SPARK-12887] Do not expose var's in TaskMetrics
Andrew Or
committed
[SPARK-12804][ML] Fix LogisticRegression with FitIntercept on all same label training data

feynmanliang
authored and
DB Tsai
committed
[SPARK-12867][SQL] Nullability of Intersect can be stricter

gatorsmile
authored and
liancheng
committed
[SPARK-12560][SQL] SqlTestUtils.stripSparkFilter needs to copy utf8strings

squito
authored and
liancheng
committed
[SPARK-12816][SQL] De-alias type when generating schemas

jodersky
authored and
marmbrus
committed
[SPARK-11295] Add packages to JUnit output for Python tests

gliptak
authored and
mengxr
committed
[BUILD] Runner for spark packages

marmbrus
authored and
rxin
committed
[SPARK-2750][WEB UI] Add https support to the Web UI

scwf
authored and
Marcelo Vanzin
committed
[SPARK-9716][ML] BinaryClassificationEvaluator should accept Double prediction column

BenFradet
authored and
jkbradley
committed

Commits on Jan 20, 2016

[SPARK-12770][SQL] Implement rules for branch elimination for CaseWhen
rxin
committed
[SPARK-12168][SPARKR] Add automated tests for conflicted function in R

felixcheung
authored and
shivaram
committed
[SPARK-12337][SPARKR] Implement dropDuplicates() method of DataFrame in SparkR.

Sun Rui
authored and
shivaram
committed
Revert "[SPARK-11295] Add packages to JUnit output for Python tests"
mengxr
committed
[SPARK-12232][SPARKR] New R API for read.table to avoid name conflict

felixcheung
authored and
shivaram
committed
[SPARK-12871][SQL] Support to specify the option for compression codec.

HyukjinKwon
authored and
rxin
committed
[SPARK-12912][SQL] Add a test suite for EliminateSubQueries
rxin
committed
[SPARK-12881] [SQL] subexpress elimination in mutable projection

Davies Liu
authored and
davies
committed
[SPARK-6519][ML] Add spark.ml API for bisecting k-means

yu-iskw
authored and
mengxr
committed
[SPARK-11295][PYSPARK] Add packages to JUnit output for Python tests

gliptak
authored and
mengxr
committed
[SPARK-12230][ML] WeightedLeastSquares.fit() should handle division by zero properly if standard deviation of target variable is zero.

iyounus
authored and
mengxr
committed
[SPARK-12925][SQL] Improve HiveInspectors.unwrap for StringObjectIns…

rbalamohan
authored and
rxin
committed
[SPARK-12898] Consider having dummyCallSite for HiveTableScan

rbalamohan
authored and
rxin
committed
[SPARK-10263][ML] Add @Since annotation to ml.param and ml.*

Takahashi Hiroshi
authored and
mengxr
committed
[SPARK-12847][CORE][STREAMING] Remove StreamingListenerBus and post all Streaming events to the same thread as Spark events

zsxwing
authored and
tdas
committed
[SPARK-7799][SPARK-12786][STREAMING] Add "streaming-akka" project

zsxwing
authored and
tdas
committed
[SPARK-12616][SQL] Making Logical Operator `Union` Support Arbitrary Number of Children

gatorsmile
authored and
rxin
committed
[SPARK-12888][SQL] benchmark the new hash expression

cloud-fan
authored and
rxin
committed
[SPARK-12848][SQL] Change parsed decimal literal datatype from Double to Decimal

hvanhovell
authored and
rxin
committed
[SPARK-12797] [SQL] Generated TungstenAggregate (without grouping keys)

Davies Liu
authored and
davies
committed

Commits on Jan 21, 2016

Commits on Jan 22, 2016

[SPARK-12908][ML] Add warning message for LogisticRegression for potential converge issue
DB Tsai
committed
[SPARK-12747][SQL] Use correct type name for Postgres JDBC's real array

viirya
authored and
rxin
committed
[SPARK-12960] [PYTHON] Some examples are missing support for python2

markgrover
authored and
davies
committed
[SPARK-12959][SQL] Writing Bucketed Data with Disabled Bucketing in SQLConf

gatorsmile
authored and
rxin
committed
[SPARK-12629][SPARKR] Fixes for DataFrame saveAsTable method

NarineK
authored and
shivaram
committed
[HOTFIX][BUILD][TEST-MAVEN] Remove duplicate dependency
zsxwing
committed

Commits on Jan 23, 2016

Commits on Jan 24, 2016

Commits on Jan 25, 2016

[SPARK-12624][PYSPARK] Checks row length when converting Java arrays to Python rows

liancheng
authored and
yhuai
committed
[SPARK-12901][SQL] Refactor options for JSON and CSV datasource (not case class and same format).

HyukjinKwon
authored and
rxin
committed
[SPARK-12932][JAVA API] improved error message for java type inference failure

andygrove
authored and
srowen
committed
[SPARK-12755][CORE] Stop the event logger before the DAG scheduler

Michael Allman
authored and
srowen
committed
[SPARK-11965][ML][DOC] Update user guide for RFormula feature interactions

yanboliang
authored and
mengxr
committed
Closes #10879
mengxr
committed
[SPARK-12149][WEB UI] Executor UI improvement suggestions - Color UI

ajbozarth
authored and
Tom Graves
committed
[SPARK-12902] [SQL] visualization for generated operators

Davies Liu
authored and
davies
committed
[SPARK-12901][SQL][HOT-FIX] Fix scala 2.11 compilation.
yhuai
committed
[SPARK-12975][SQL] Throwing Exception when Bucketing Columns are part of Partitioning Columns

gatorsmile
authored and
marmbrus
committed
[SPARK-12905][ML][PYSPARK] PCAModel return eigenvalues for PySpark

yanboliang
authored and
mengxr
committed
[SPARK-12934][SQL] Count-min sketch serialization

liancheng
authored and
rxin
committed

Commits on Jan 26, 2016

[SPARK-12879] [SQL] improve the unsafe row writing framework

cloud-fan
authored and
davies
committed
[SPARK-12936][SQL] Initial bloom filter implementation

cloud-fan
authored and
rxin
committed
[SPARK-12934] use try-with-resources for streams

tedyu
authored and
liancheng
committed
[SPARK-11922][PYSPARK][ML] Python api for ml.feature.quantile discretizer

holdenk
authored and
jkbradley
committed
[SPARK-12834] Change ser/de of JavaArray and JavaList

yinxusen
authored and
jkbradley
committed
[SPARK-10086][MLLIB][STREAMING][PYSPARK] ignore StreamingKMeans test in PySpark for now
mengxr
committed
[SQL][MINOR] A few minor tweaks to CSV reader.
rxin
committed
[SPARK-12937][SQL] bloom filter serialization

cloud-fan
authored and
rxin
committed
[SPARK-12961][CORE] Prevent snappy-java memory leak

viirya
authored and
srowen
committed
[SPARK-3369][CORE][STREAMING] Java mapPartitions Iterator->Iterable is inconsistent with Scala's Iterator->Iterator
srowen
committed
[SPARK-10911] Executors should System.exit on clean shutdown.

zhuol
authored and
Tom Graves
committed
[SPARK-12682][SQL] Add support for (optionally) not storing tables in hive metadata format

sameeragarwal
authored and
yhuai
committed
[SPARK-7799][STREAMING][DOCUMENT] Add the linking and deploying instructions for streaming-akka project

zsxwing
authored and
tdas
committed
[SPARK-11923][ML] Python API for ml.feature.ChiSqSelector

yinxusen
authored and
jkbradley
committed
[SPARK-12952] EMLDAOptimizer initialize() should return EMLDAOptimizer other than its parent class

yinxusen
authored and
jkbradley
committed
[SPARK-8725][PROJECT-INFRA] Test modules in topologically-sorted order in dev/run-tests
JoshRosen
committed
[SQL] Minor Scaladoc format fix
liancheng
committed
[SPARK-12993][PYSPARK] Remove usage of ADD_FILES in pyspark

zjffdu
authored and
rxin
committed
[SPARK-10509][PYSPARK] Reduce excessive param boiler plate code

holdenk
authored and
jkbradley
committed

Commits on Jan 27, 2016

[SPARK-12614][CORE] Don't throw non fatal exception from ask
zsxwing
committed
[SPARK-11622][MLLIB] Make LibSVMRelation extends HadoopFsRelation and…

zjffdu
authored and
mengxr
committed
[SPARK-12854][SQL] Implement complex types support in ColumnarBatch

nongli
authored and
rxin
committed
[SPARK-7780][MLLIB] intercept in logisticregressionwith lbfgs should not be regularized

holdenk
authored and
DB Tsai
committed
[SPARK-12903][SPARKR] Add covar_samp and covar_pop for SparkR

yanboliang
authored and
shivaram
committed
[SPARK-12935][SQL] DataFrame API for Count-Min Sketch

liancheng
authored and
rxin
committed
[SPARK-12728][SQL] Integrates SQL generation with native view

liancheng
authored and
yhuai
committed
[SPARK-12967][NETTY] Avoid NettyRpc error message during sparkContext shutdown

nishkamravi2
authored and
zsxwing
committed
[SPARK-12780] Inconsistency returning value of ML python models' properties

yinxusen
authored and
jkbradley
committed
[SPARK-12983][CORE][DOC] Correct metrics.properties.template

BenFradet
authored and
srowen
committed
[SPARK-1680][DOCS] Explain environment variables for running on YARN in cluster mode

weineran
authored and
srowen
committed
[SPARK-13023][PROJECT INFRA] Fix handling of root module in modules_to_test()
JoshRosen
committed
[SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with `None` triggers cryptic failure

jasoncl
authored and
yhuai
committed
[SPARK-12895][SPARK-12896] Migrate TaskMetrics to accumulators

Andrew Or
authored and
JoshRosen
committed
[SPARK-13021][CORE] Fail fast when custom RDDs violate RDD.partition's API contract

JoshRosen
authored and
yhuai
committed
[SPARK-12938][SQL] DataFrame API for Bloom filter

cloud-fan
authored and
rxin
committed
[SPARK-12865][SPARK-12866][SQL] Migrate SparkSQLParser/ExtendedHiveQlParser commands to new Parser

hvanhovell
authored and
rxin
committed
[HOTFIX] Fix Scala 2.11 compilation

Andrew Or
authored and
yhuai
committed
[SPARK-13045] [SQL] Remove ColumnVector.Struct in favor of ColumnarBatch.Row

nongli
authored and
davies
committed

Commits on Jan 28, 2016

Provide same info as in spark-submit --help

jimlohse
authored and
srowen
committed
[SPARK-12818][SQL] Specialized integral and string types for Count-min Sketch

liancheng
authored and
rxin
committed
[SPARK-12926][SQL] SQLContext to display warning message when non-sql configs are being set

tejasapatil
authored and
marmbrus
committed
[SPARK-13031] [SQL] cleanup codegen and improve test coverage

Davies Liu
authored and
davies
committed
[SPARK-9835][ML] Implement IterativelyReweightedLeastSquares solver

yanboliang
authored and
mengxr
committed
[SPARK-12401][SQL] Add integration tests for postgres enum types

maropu
authored and
liancheng
committed
[SPARK-12749][SQL] add json option to parse floating-point types as DecimalType

blbradley
authored and
rxin
committed

Commits on Jan 29, 2016

[SPARK-11955][SQL] Mark optional fields in merging schema for safely pushdowning filters in Parquet

viirya
authored and
liancheng
committed
Revert "[SPARK-13031] [SQL] cleanup codegen and improve test coverage"
davies
committed
[SPARK-12968][SQL] Implement command to set current database

viirya
authored and
rxin
committed
[SPARK-13067] [SQL] workaround for a weird scala reflection problem

cloud-fan
authored and
davies
committed
[SPARK-13050][BUILD] Scalatest tags fail build with the addition of the sketch module

ajbozarth
authored and
liancheng
committed
[SPARK-13031][SQL] cleanup codegen and improve test coverage

Davies Liu
authored and
rxin
committed
[SPARK-13032][ML][PYSPARK] PySpark support model export/import and take LinearRegression as example

yanboliang
authored and
jkbradley
committed
[SPARK-10873] Support column sort and search for History Server.

zhuol
authored and
Tom Graves
committed
[SPARK-13072] [SQL] simplify and improve murmur3 hash expression codegen

cloud-fan
authored and
davies
committed
[SPARK-12656] [SQL] Implement Intersect with Left-semi Join

gatorsmile
authored and
rxin
committed
[SPARK-12818] Polishes spark-sketch module
liancheng
committed
[SPARK-13055] SQLHistoryListener throws ClassCastException

Andrew Or
authored and
zsxwing
committed

Commits on Jan 30, 2016

[SPARK-13076][SQL] Rename ClientInterface -> HiveClient
rxin
committed
[SPARK-13096][TEST] Fix flaky verifyPeakExecutionMemorySet
Andrew Or
committed
[SPARK-13088] Fix DAG viz in latest version of chrome
Andrew Or
committed
[SPARK-13071] Coalescing HadoopRDD overwrites existing input metrics
Andrew Or
committed
[SPARK-12914] [SQL] generate aggregation with grouping keys

Davies Liu
authored and
davies
committed
[SPARK-13098] [SQL] remove GenericInternalRowWithSchema

cloud-fan
authored and
davies
committed
[SPARK-6363][BUILD] Make Scala 2.11 the default Scala version

JoshRosen
authored and
rxin
committed
[SPARK-13100][SQL] improving the performance of stringToDate method in DateTimeUtils.scala

wangyang
authored and
rxin
committed