merge upstream changes#1

Merged

nchammas merged 76 commits intonchammas:masterfrom

Sep 16, 2014

Commits on Sep 8, 2014

[SPARK-3086] [SPARK-3043] [SPARK-3156] [mllib] DecisionTree aggregation improvements

jkbradley
authored and
mengxr
committed
SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within.

ScrapCodes
authored and
andrewor14
committed
SPARK-2978. Transformation with MR shuffle semantics

sryza
authored and
mateiz
committed
Provide a default PYSPARK_PYTHON for python/run_tests

Matthew Farrellee
authored and
JoshRosen
committed
[SQL] Minor edits to sql programming guide.

hcook
authored and
marmbrus
committed
[SPARK-3417] Use new-style classes in PySpark

mrocklin
authored and
marmbrus
committed
[SPARK-3019] Pluggable block transfer interface (BlockTransferService)
rxin
committed
[SPARK-3349][SQL] Output partitioning of limit should not be inherited from child

Eric Liang
authored and
marmbrus
committed

Commits on Sep 9, 2014

[SPARK-3443][MLLIB] update default values of tree:
mengxr
committed
SPARK-3423: [SQL] Implement BETWEEN for SQLParser

willb
authored and
marmbrus
committed
[SPARK-3414][SQL] Stores analyzed logical plan when registering a temp table

liancheng
authored and
marmbrus
committed
[SPARK-3329][SQL] Don't depend on Hive SET pair ordering in tests.

willb
authored and
marmbrus
committed
SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors

markhamstra
authored and
andrewor14
committed
[Build] Removed -Phive-thriftserver since this profile has been removed

liancheng
authored and
rxin
committed
[Docs] actorStream storageLevel default is MEMORY_AND_DISK_SER_2

melrief
authored and
rxin
committed
[SPARK-3455] [SQL] **HOT FIX** Fix the unit test failure

chenghao-intel
authored and
rxin
committed
SPARK-3422. JavaAPISuite.getHadoopInputSplits isn't used anywhere.

sryza
authored and
rxin
committed
SPARK-3404 [BUILD] SparkSubmitSuite fails with "spark-submit exits with code 1"

srowen
authored and
rxin
committed
[SPARK-3193]output errer info when Process exit code is not zero in test suite

scwf
authored and
andrewor14
committed
Minor - Fix trivial compilation warnings.

ScrapCodes
authored and
rxin
committed
[SPARK-3176] Implement 'ABS and 'LAST' for sql

xinyunh
authored and
marmbrus
committed

Commits on Sep 10, 2014

[SPARK-3448][SQL] Check for null in SpecificMutableRow.update

liancheng
authored and
marmbrus
committed
[SPARK-3458] enable python "with" statements for SparkContext

Matthew Farrellee
authored and
rxin
committed
[SPARK-3395] [SQL] DSL sometimes incorrectly reuses attribute ids, breaking queries

Eric Liang
authored and
marmbrus
committed
[SPARK-3286] - Cannot view ApplicationMaster UI when Yarn’s url scheme i...

Benoy Antony
authored and
tgravescs
committed
[SPARK-3362][SQL] Fix resolution for casewhen with nulls.

adrian-wang
authored and
marmbrus
committed
[SPARK-3363][SQL] Type Coercion should promote null to all other types.

adrian-wang
authored and
marmbrus
committed
[HOTFIX] Fix scala style issue introduced by #2276.
JoshRosen
committed
SPARK-1713. Use a thread pool for launching executors.

sryza
authored and
tgravescs
committed
[SPARK-2096][SQL] Correctly parse dot notations

cloud-fan
authored and
marmbrus
committed
[SPARK-3411] Improve load-balancing of concurrently-submitted drivers across workers

WangTaoTheTonic
authored and
andrewor14
committed
[SPARK-2207][SPARK-3272][MLLib]Add minimum information gain and minimum instances per node as training parameters for decision tree.

qiping.lqp
authored and
mengxr
committed

Commits on Sep 11, 2014

[SQL] Add test case with workaround for reading partitioned Avro files
marmbrus
committed
[SPARK-3447][SQL] Remove explicit conversion with JListWrapper to avoid NPE
marmbrus
committed
[SPARK-2781][SQL] Check resolution of LogicalPlans in Analyzer.

staple
authored and
marmbrus
committed
[SPARK-2140] Updating heap memory calculation for YARN stable and alpha.

Chris Cope
authored and
tgravescs
committed
[SPARK-3047] [PySpark] add an option to use str in textFileRDD

davies
authored and
JoshRosen
committed
[SPARK-2917] [SQL] Avoid table creation in logical plan analyzing for CTAS

chenghao-intel
authored and
marmbrus
committed
[SPARK-3390][SQL] sqlContext.jsonRDD fails on a complex structure of JSON array and JSON object nesting

yhuai
authored and
marmbrus
committed

Commits on Sep 12, 2014

[Spark-3490] Disable SparkUI for tests
andrewor14
committed
[SPARK-3429] Don't include the empty string "" as a defaultAclUser

ash211
authored and
andrewor14
committed
SPARK-3462 push down filters and projections into Unions

Cody Koeninger
authored and
marmbrus
committed
SPARK-2482: Resolve sbt warnings during build

witgo
authored and
andrewor14
committed
[SPARK-3465] fix task metrics aggregation in local mode

davies
authored and
andrewor14
committed
[SPARK-3160] [SPARK-3494] [mllib] DecisionTree: eliminate pre-allocated nodes, parentImpurities arrays. Memory calc bug fix.

jkbradley
authored and
mengxr
committed
[SPARK-2558][DOCS] Add --queue example to YARN doc

kramimus
authored and
tgravescs
committed
[PySpark] Add blank line so that Python RDD.top() docstring renders correctly

rnowling
authored and
JoshRosen
committed
[SPARK-3481] [SQL] Eliminate the error log in local Hive comparison test

chenghao-intel
authored and
marmbrus
committed
MAINTENANCE: Automated closing of pull requests.
pwendell
committed
[SPARK-3427] [GraphX] Avoid active vertex tracking in static PageRank

ankurdave
authored and
rxin
committed
SPARK-3014. Log a more informative messages in a couple failure scenario...

sryza
authored and
tgravescs
committed
[SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.

Marcelo Vanzin
authored and
pwendell
committed

Commits on Sep 13, 2014

[SPARK-3456] YarnAllocator on alpha can lose container requests to RM
tgravescs
committed
[SPARK-3094] [PySpark] compatitable with PyPy

davies
authored and
JoshRosen
committed
[SPARK-3500] [SQL] use JavaSchemaRDD as SchemaRDD._jschema_rdd

davies
authored and
JoshRosen
committed
[SPARK-3515][SQL] Moves test suite setup code to beforeAll rather than in constructor

liancheng
authored and
marmbrus
committed
[SPARK-3469] Make sure all TaskCompletionListener are called even with failures
rxin
committed
[SQL][Docs] Update SQL programming guide to show the correct default value of containsNull in an ArrayType

yhuai
authored and
rxin
committed
SPARK-3470 [CORE] [STREAMING] Add Closeable / close() to Java context objects

srowen
authored and
rxin
committed
Proper indent for the previous commit.
rxin
committed
[SQL] [Docs] typo fixes

nchammas
authored and
marmbrus
committed
[SPARK-3481][SQL] Removes the evil MINOR HACK

liancheng
authored and
marmbrus
committed
[SPARK-3294][SQL] Eliminates boxing costs from in-memory columnar storage

liancheng
authored and
marmbrus
committed
[SQL] Decrease partitions when testing
marmbrus
committed
[SPARK-3030] [PySpark] Reuse Python worker

davies
authored and
JoshRosen
committed

Commits on Sep 14, 2014

[SPARK-3463] [PySpark] aggregate and show spilled bytes in Python

davies
authored and
JoshRosen
committed

Commits on Sep 15, 2014

SPARK-3039: Allow spark to be built using avro-mapred for hadoop2

bbossy
authored and
pwendell
committed
[SPARK-3452] Maven build should skip publishing artifacts people shouldn...

ScrapCodes
authored and
pwendell
committed
[SPARK-3410] The priority of shutdownhook for ApplicationMaster should not be integer literal

sarutak
authored and
tgravescs
committed
[SPARK-3425] do not set MaxPermSize for OpenJDK 1.8

Matthew Farrellee
authored and
pwendell
committed
[SPARK-3518] Remove wasted statement in JsonProtocol

sarutak
authored and
pwendell
committed
[SPARK-2714] DAGScheduler logs jobid when runJob finishes

YanTangZhai
authored and
andrewor14
committed

Commits on Sep 16, 2014

[SPARK-3396][MLLIB] Use SquaredL2Updater in LogisticRegressionWithSGD

BigCrunsh
authored and
mengxr
committed
[MLlib] Update SVD documentation in IndexedRowMatrix

rezazadeh
authored and
mengxr
committed
[SPARK-3516] [mllib] DecisionTree: Add minInstancesPerNode, minInfoGain params to example and Python API

qiping.lqp
authored and
mengxr
committed
[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6

davies
authored and
JoshRosen
committed