[SPARK-7943][SQL][WIP]support DataFrame created by hiveContext can create table to specific database by saveAstable#6494

Closed

baishuo wants to merge 289 commits intoapache:masterfrom

baishuo:SPARK-7943-20150529

This pull request is big! We're only showing the most recent 250 commits

Commits on May 31, 2015

[SQL] [MINOR] Adds @deprecated Scaladoc entry for SchemaRDD

liancheng
authored and
rxin
committed
[SPARK-7975] Add style checker to disallow overriding equals covariantly.
rxin
committed
[SPARK-3850] Trim trailing spaces for core.
rxin
committed
[SPARK-3850] Trim trailing spaces for examples/streaming/yarn.
rxin
committed
[SPARK-3850] Trim trailing spaces for SQL.
rxin
committed
[SPARK-7979] Enforce structural type checker.
rxin
committed
[MINOR] Add license for dagre-d3 and graphlib-dot

zsxwing
authored and
Andrew Or
committed
[SPARK-3850] Trim trailing spaces for MLlib.
rxin
committed
[SPARK-7949] [MLLIB] [DOC] update document with some missing save/load

hhbyyh
authored and
jkbradley
committed
[SPARK-3850] Turn style checker on for trailing whitespaces.
rxin
committed
[SPARK-7227] [SPARKR] Support fillna / dropna in R DataFrame.

Sun Rui
authored and
shivaram
committed
[MINOR] Enable PySpark SQL readerwriter and window tests

JoshRosen
authored and
rxin
committed

Commits on Jun 1, 2015

[SPARK-7986] Split scalastyle config into 3 sections.
rxin
committed
[SPARK-7978] [SQL] [PYSPARK] DecimalType should not be singleton

Davies Liu
authored and
rxin
committed
[SPARK-7952][SPARK-7984][SQL] equality check between boolean type and numeric type is broken.

cloud-fan
authored and
rxin
committed
Update README to include DataFrames and zinc.
rxin
committed
[DOC] Minor modification to Streaming docs with regards to parallel data receiving

nishkamravi2
authored and
srowen
committed
[SPARK-7497] [PYSPARK] [STREAMING] fix streaming flaky tests

Davies Liu
authored and
tdas
committed
[SPARK-7584] [MLLIB] User guide for VectorAssembler
mengxr
committed

Commits on Jun 2, 2015

Commits on Jun 3, 2015

[SQL] [TEST] [MINOR] Follow-up of PR #6493, use Guava API to ensure Java 6 friendliness

liancheng
authored and
Andrew Or
committed
[SPARK-7387] [ML] [DOC] CrossValidator example code in Python

Ram Sriharsha
authored and
jkbradley
committed
[SPARK-7547] [ML] Scala Example code for ElasticNet

DB Tsai
authored and
jkbradley
committed
[SPARK-7691] [SQL] Refactor CatalystTypeConverter to use type-specific row accessors

JoshRosen
authored and
rxin
committed
add test case
baishuo
committed
[SPARK-8053] [MLLIB] renamed scalingVector to scalingVec

jkbradley
authored and
mengxr
committed
[MINOR] make the launcher project name consistent with others

WangTaoTheTonic
authored and
pwendell
committed
correct style
baishuo
committed
[SPARK-8043] [MLLIB] [DOC] update NaiveBayes and SVM examples in doc

hhbyyh
authored and
mengxr
committed
[SPARK-8032] [PYSPARK] Make version checking for NumPy in MLlib more robust

MechCoder
authored and
mengxr
committed
[SPARK-8060] Improve DataFrame Python test coverage and documentation.
rxin
committed
[SPARK-7562][SPARK-6444][SQL] Improve error reporting for expression data type mismatch

cloud-fan
authored and
rxin
committed
[SPARK-7983] [MLLIB] Add require for one-based indices in loadLibSVMFile

hhbyyh
authored and
srowen
committed
[SPARK-7973] [SQL] Increase the timeout of two CliSuite tests.
yhuai
committed
[SPARK-7801] [BUILD] Updating versions to SPARK 1.5.0
pwendell
committed
[SPARK-7980] [SQL] Support SQLContext.range(end)

animesh
authored and
rxin
committed
[SPARK-7161] [HISTORY SERVER] Provide REST api to download event logs fro...

harishreedharan
authored and
squito
committed
[SPARK-8063] [SPARKR] Spark master URL conflict between MASTER env variable and --master command line option.

Sun Rui
authored and
shivaram
committed
[SPARK-8074] Parquet should throw AnalysisException during setup for data type/name related failures.
rxin
committed
Update documentation for [SPARK-7980] [SQL] Support SQLContext.range(end)
rxin
committed
[SPARK-8054] [MLLIB] Added several Java-friendly APIs + unit tests
jkbradley
committed
[MINOR] [UI] Improve confusing message on log page
Andrew Or
committed
[SPARK-8083] [MESOS] Use the correct base path in mesos driver page.

tnachen
authored and
Andrew Or
committed
[SPARK-8059] [YARN] Wake up allocation thread when new requests arrive.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-8001] [CORE] Make AsynchronousListenerBus.waitUntilEmpty throw TimeoutException if timeout

zsxwing
authored and
Andrew Or
committed
[SPARK-7989] [CORE] [TESTS] Fix flaky tests in ExternalShuffleServiceSuite and SparkListenerWithClusterSuite

zsxwing
authored and
Andrew Or
committed
[HOTFIX] Fix Hadoop-1 build caused by #5792.

harishreedharan
authored and
Andrew Or
committed
[SPARK-3674] [EC2] Clear SPARK_WORKER_INSTANCES when using YARN

shivaram
authored and
Andrew Or
committed
[SPARK-8051] [MLLIB] make StringIndexerModel silent if input column does not exist

mengxr
authored and
jkbradley
committed
[SPARK-6164] [ML] CrossValidatorModel should keep stats from fitting

leahmcguire
authored and
jkbradley
committed
[HOTFIX] [TYPO] Fix typo in #6546
Andrew Or
committed
[HOTFIX] History Server API docs error fix.

harishreedharan
authored and
Andrew Or
committed
[SPARK-8088] don't attempt to lower number of executors by 0

ryan-williams
authored and
Andrew Or
committed

Commits on Jun 4, 2015

[SPARK-8084] [SPARKR] Make SparkR scripts fail on error
shivaram
committed
[BUILD] Increase Jenkins test timeout
Andrew Or
committed
[BUILD] Use right branch when checking against Hive
Andrew Or
committed
[BUILD] Fix Maven build for Kinesis
Andrew Or
committed
MAINTENANCE: Automated closing of pull requests.
pwendell
committed
modify testcase
baishuo
committed
Fix maxTaskFailures comment

darabos
authored and
srowen
committed
[SPARK-7956] [SQL] Use Janino to compile SQL expressions into bytecode
Davies Liu
committed
[SPARK-7969] [SQL] Added a DataFrame.drop function that accepts a Column reference.

dusenberrymw
authored and
rxin
committed
[SPARK-7743] [SQL] Parquet 1.7

Thomas Omans
authored and
rxin
committed
[SPARK-8027] [SPARKR] Move man pages creation to install-dev.sh
shivaram
committed
[SPARK-6909][SQL] Remove Hive Shim code

Cheolsoo Park
authored and
rxin
committed
Fixed style issues for [SPARK-6909][SQL] Remove Hive Shim code.
rxin
committed
[SPARK-7440][SQL] Remove physical Distinct operator in favor of Aggregate
rxin
committed
[SPARK-8098] [WEBUI] Show correct length of bytes on log page

carsonwang
authored and
tdas
committed

Commits on Jun 5, 2015

[SPARK-8106] [SQL] Set derby.system.durability=test to speed up Hive compatibility tests

JoshRosen
authored and
rxin
committed
[SPARK-8114][SQL] Remove some wildcard import on TestSQLContext._
rxin
committed
[SPARK-8116][PYSPARK] Allow sc.range() to take a single argument.

belisarius222
authored and
rxin
committed
[DOC][Minor]Specify the common sources available for collecting

yjshen
authored and
srowen
committed
[MINOR] remove unused interpolation var in log message

srowen
authored and
rxin
committed
[MINOR] [BUILD] Change link to jenkins builds on github.

Marcelo Vanzin
authored and
srowen
committed
[MINOR] [BUILD] Use custom temp directory during build.

Marcelo Vanzin
authored and
srowen
committed
[STREAMING] Update streaming-kafka-integration.md

Akhil Das
authored and
srowen
committed
[SPARK-6324] [CORE] Centralize handling of script usage messages.

Marcelo Vanzin
authored and
srowen
committed
[SQL] Simplifies binary node pattern matching
liancheng
committed
[SPARK-8085] [SPARKR] Support user-specified schema in read.df
shivaram
committed
Revert "[MINOR] [BUILD] Use custom temp directory during build."
Andrew Or
committed
[SPARK-8099] set executor cores into system in yarn-cluster mode

XuTingjun
authored and
sryza
committed
[SPARK-7699] [CORE] Lazy start the scheduler for dynamic allocation

jerryshao
authored and
sryza
committed
[SPARK-8112] [STREAMING] Fix the negative event count issue

zsxwing
authored and
tdas
committed
[SPARK-7747] [SQL] [DOCS] spark.sql.planner.externalSort

lucamartinetti
authored and
yhuai
committed
[SPARK-7991] [PySpark] Adding support for passing lists to describe.

amey
authored and
rxin
committed
[SPARK-8114][SQL] Remove some wildcard import on TestSQLContext._ cont'd.
rxin
committed

Commits on Jun 7, 2015

[SPARK-7169] [CORE] Allow metrics system to be configured through SparkConf.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-8136] [YARN] Fix flakiness in YarnClusterSuite.

harishreedharan
authored and
Andrew Or
committed
[SPARK-7955] [CORE] Ensure executors with cached RDD blocks are not re…

harishreedharan
authored and
Andrew Or
committed
[SPARK-8004][SQL] Enclose column names by JDBC Dialect

viirya
authored and
rxin
committed
[SPARK-8145] [WEBUI] Trigger a double click on the span to show full job description.

979969786
authored and
Andrew Or
committed
[SPARK-8141] [SQL] Precompute datatypes for partition columns and reuse it

viirya
authored and
liancheng
committed
[SPARK-8146] DataFrame Python API: Alias replace in df.na
rxin
committed
[SPARK-8118] [SQL] Mutes noisy Parquet log output reappeared after upgrading Parquet to 1.7.0
liancheng
committed
[SPARK-7042] [BUILD] use the standard akka artifacts with hadoop-2.x

Konstantin Shaposhnikov
authored and
srowen
committed
[DOC] [TYPO] Fix typo in standalone deploy scripts description

yjshen
authored and
srowen
committed
[SPARK-8004][SQL] Quote identifier in JDBC data source.
rxin
committed
[SPARK-7952][SQL] use internal Decimal instead of java.math.BigDecimal

cloud-fan
authored and
rxin
committed
[SPARK-7733] [CORE] [BUILD] Update build, code to use Java 7 for 1.5.0+
srowen
committed
[SPARK-2808] [STREAMING] [KAFKA] cleanup tests from

koeninger
authored and
srowen
committed
[SPARK-8117] [SQL] Push codegen implementation into each Expression

Davies Liu
authored and
rxin
committed

Commits on Jun 8, 2015

[SPARK-8149][SQL] Break ExpressionEvaluationSuite down to multiple files
rxin
committed
[SPARK-8154][SQL] Remove Term/Code type aliases in code generation.
rxin
committed
[SPARK-4761] [DOC] [SQL] kryo default setting in SQL Thrift server

adrian-wang
authored and
rxin
committed
[SPARK-7705] [YARN] Cleanup of .sparkStaging directory fails if application is killed

Sephiroth-Lin
authored and
srowen
committed
[SPARK-7939] [SQL] Add conf to enable/disable partition column type inference

viirya
authored and
liancheng
committed
[SPARK-8126] [BUILD] Use custom temp directory during build.

Marcelo Vanzin
authored and
srowen
committed
[SPARK-8140] [MLLIB] Remove empty model check in StreamingLinearAlgorithm

MechCoder
authored and
srowen
committed
[SMALL FIX] Return null if catch EOFException

Mingfei
authored and
srowen
committed
[MINOR] change new Exception to IllegalArgumentException

adrian-wang
authored and
rxin
committed
[SPARK-8158] [SQL] several fix for HiveShim

adrian-wang
authored and
rxin
committed
[SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRelation job initialization for Hadoop 1.x

liancheng
authored and
yhuai
committed
[SQL][minor] remove duplicated cases in `DecimalPrecision`

cloud-fan
authored and
rxin
committed
[SPARK-8148] Do not use FloatType in partition column inference.
rxin
committed

Commits on Jun 9, 2015

[SPARK-8162] [HOTFIX] Fix NPE in spark-shell
Andrew Or
committed
[SPARK-8168] [MLLIB] Add Python friendly constructor to PipelineModel

mengxr
authored and
DB Tsai
committed
[SPARK-6820] [SPARKR] Convert NAs to null type in SparkR DataFrames

hqzizania
authored and
shivaram
committed
[SPARK-7990][SQL] Add methods to facilitate equi-join on multiple joining keys

viirya
authored and
rxin
committed
[SPARK-8101] [CORE] Upgrade netty to avoid memory leak accord to netty #3837 issues
srowen
committed
[SPARK-7886] Add built-in expressions to FunctionRegistry.

rxin
authored and
liancheng
committed
[STREAMING] [DOC] Remove duplicated description about WAL

sarutak
authored and
srowen
committed
[SPARK-8140] [MLLIB] Remove construct to get weights in StreamingLinearAlgorithm

MechCoder
authored and
srowen
committed
[SPARK-8274] [DOCUMENTATION-MLLIB] Fix wrong URLs in MLlib Frequent Pattern Mining Documentation

FavioVazquez
authored and
srowen
committed
[MINOR] [UI] DAG visualization: trim whitespace from input
Andrew Or
committed
[SPARK-6511] [DOCUMENTATION] Explain how to use Hadoop provided builds
pwendell
committed

Commits on Jun 10, 2015

[SPARK-7792] [SQL] HiveContext registerTempTable not thread safe

navis
authored and
rxin
committed
[SPARK-7886] Use FunctionRegistry for built-in expressions in HiveContext.
rxin
committed
[SPARK-7886] Added unit test for HAVING aggregate pushdown.

rxin
authored and
liancheng
committed
[SPARK-8215] [SPARK-8212] [SQL] add leaf math expression for e and pi

adrian-wang
authored and
rxin
committed
[SPARK-7996] Deprecate the developer api SparkEnv.actorSystem

Ilya Ganelin
authored and
rxin
committed
[SQL] [MINOR] Fixes a minor Java example error in SQL programming guide

liancheng
authored and
rxin
committed
[SPARK-5479] [YARN] Handle --py-files correctly in YARN.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-8282] [SPARKR] Make number of threads used in RBackend configurable

falaki
authored and
Andrew Or
committed
[SPARK-7756] CORE RDDOperationScope fix for IBM Java

a-roberts
authored and
Andrew Or
committed
[SPARK-7527] [CORE] Fix createNullValue to return the correct null values and REPL mode detection

zsxwing
authored and
Andrew Or
committed
[SPARK-7261] [CORE] Change default log level to WARN in the REPL

zsxwing
authored and
Andrew Or
committed
[SPARK-8290] spark class command builder need read SPARK_JAVA_OPTS and SPARK_DRIVER_MEMORY properly

WangTaoTheTonic
authored and
Andrew Or
committed
[SPARK-8273] Driver hangs up when yarn shutdown in client mode

WangTaoTheTonic
authored and
Andrew Or
committed
[SPARK-2774] Set preferred locations for reduce tasks

shivaram
authored and
kayousterhout
committed
[SPARK-8200] [MLLIB] Check for empty RDDs in StreamingLinearAlgorithm

pparkkin
authored and
srowen
committed
[SPARK-8189] [SQL] use Long for TimestampType in SQL

Davies Liu
authored and
rxin
committed

Commits on Jun 11, 2015

[SPARK-8285] [SQL] CombineSum should be calculated as unlimited decimal first

navis
authored and
rxin
committed
[SPARK-8164] transformExpressions should support nested expression sequence

cloud-fan
authored and
rxin
committed
[SPARK-8248][SQL] string function: length

chenghao-intel
authored and
rxin
committed
[SPARK-8217] [SQL] math function log2

adrian-wang
authored and
rxin
committed
[HOTFIX] Adding more contributor name bindings
pwendell
committed
[HOTFIX] Fixing errors in name mappings
pwendell
committed
[SPARK-8289] Specify stack size for consistency with Java tests - resolves test failures

a-roberts
authored and
srowen
committed
[SPARK-6411] [SQL] [PySpark] support date/datetime with timezone in Python

Davies Liu
authored and
rxin
committed
[SPARK-8305] [SPARK-8190] [SQL] improve codegen

Davies Liu
authored and
rxin
committed
[SPARK-8310] [EC2] Updates the master branch EC2 versions
shivaram
committed
[SPARK-7915] [SQL] Support specifying the column list for target table in CTAS

chenghao-intel
authored and
marmbrus
committed
[SPARK-7444] [TESTS] Eliminate noisy css warn/error logs for UISeleniumSuite

zsxwing
authored and
marmbrus
committed
[SPARK-6511] [docs] Fix example command in hadoop-provided docs.

Marcelo Vanzin
authored and
rxin
committed
[SPARK-8286] Rewrite UTF8String in Java and move it into unsafe package.
rxin
committed

Commits on Jun 12, 2015

[SPARK-7824] [SQL] Collapse operator reordering and constant folding into a single batch.

pzzs
authored and
marmbrus
committed
[SQL] Miscellaneous SQL/DF expression changes.
rxin
committed
[SPARK-7158] [SQL] Fix bug of cached data cannot be used in collect() after cache()

chenghao-intel
authored and
marmbrus
committed
[SPARK-8317] [SQL] Do not push sort into shuffle in Exchange operator

JoshRosen
authored and
marmbrus
committed
[SPARK-7862] [SQL] Fix the deadlock in script transformation for stderr

zhichao-li
authored and
marmbrus
committed
[SPARK-6566] [SQL] Related changes for newer parquet version

Yash Datta
authored and
liancheng
committed
[SQL] [MINOR] correct semanticEquals logic

cloud-fan
authored and
liancheng
committed
[SPARK-8322] [EC2] Added spark 1.4.0 into the VALID_SPARK_VERSIONS and…

Mark Smith
authored and
shivaram
committed
[SPARK-7993] [SQL] Improved DataFrame.show() output

akhilthatipamula
authored and
rxin
committed
[SPARK-8330] DAG visualization: trim whitespace from input
Andrew Or
committed
[SPARK-7284] [STREAMING] Updated streaming documentation
tdas
committed

Commits on Jun 13, 2015

[SPARK-8314][MLlib] improvement in performance of MLUtils.appendBias

rogermenezes
authored and
DB Tsai
committed
[SPARK-7186] [SQL] Decouple internal Row from external Row

Davies Liu
authored and
rxin
committed
[SPARK-8329][SQL] Allow _ in DataSource options

marmbrus
authored and
rxin
committed
[SPARK-7897] Improbe type for jdbc/"unsigned bigint"

rtreffer
authored and
rxin
committed
[SPARK-8346] [SQL] Use InternalRow instread of catalyst.InternalRow

Davies Liu
authored and
rxin
committed
[SPARK-8319] [CORE] [SQL] Update logic related to key orderings in shuffle dependencies
JoshRosen
committed
[SPARK-8052] [SQL] Use java.math.BigDecimal for casting String to Decimal instead of using toDouble

viirya
authored and
marmbrus
committed

Commits on Jun 14, 2015

[SPARK-8347][SQL] Add unit tests for abs.
rxin
committed
[SPARK-8349] [SQL] Use expression constructors (rather than apply) in FunctionRegistry

rxin
authored and
marmbrus
committed
[Spark-8343] [Streaming] [Docs] Improve Spark Streaming Guides.

dusenberrymw
authored and
rxin
committed
[SPARK-8342][SQL] Fix Decimal setOrNull

viirya
authored and
rxin
committed
[SPARK-8354] [SQL] Fix off-by-factor-of-8 error when allocating scratch space in UnsafeFixedWidthAggregationMap
JoshRosen
committed
[SPARK-8358] [SQL] Wait for child resolution when resolving generators
marmbrus
committed
[SPARK-8362] [SQL] Add unit tests for +, -, *, /, %

rxin
authored and
marmbrus
committed
fix read/write mixup

hoffmann
authored and
rxin
committed
[SPARK-8065] [SQL] Add support for Hive 0.14 metastores

Marcelo Vanzin
authored and
marmbrus
committed

Commits on Jun 15, 2015

[SPARK-8316] Upgrade to Maven 3.3.3

nchammas
authored and
srowen
committed
[SPARK-8350] [R] Log R unit test output to "unit-tests.log"

andrewor14
authored and
shivaram
committed
[SPARK-6583] [SQL] Support aggregate functions in ORDER BY

watermen
authored and
marmbrus
committed

Commits on Jun 16, 2015

SPARK-8336 Fix NullPointerException with functions.rand()

tedyu
authored and
rxin
committed
[SPARK-7184] [SQL] enable codegen by default

Davies Liu
authored and
rxin
committed
[SPARK-8367] [STREAMING] Add a limit for 'spark.streaming.blockInterval` since a data loss bug.

SaintBacchus
authored and
srowen
committed
[SPARK-8129] [CORE] [Sec] Pass auth secrets to executors via env variables

kanzhang
authored and
srowen
committed
[SPARK-8387] [WEBUI] Only show 4096 bytes content for executor log instead of show all

suyanNone
authored and
srowen
committed
[SPARK-DOCS] [SPARK-SQL] Update sql-programming-guide.md

moutai
authored and
srowen
committed
[SQL] [DOC] improved a comment

radek1st
authored and
srowen
committed
[SPARK-8126] [BUILD] Make sure temp dir exists when running tests.

Marcelo Vanzin
authored and
srowen
committed
[SPARK-7916] [MLLIB] MLlib Python doc parity check for classification and regression

yanboliang
authored and
jkbradley
committed
[SPARK-8156] [SQL] create table to specific database by 'use dbname'

baishuo
authored and
marmbrus
committed

Commits on Jun 17, 2015

[SPARK-8220][SQL]Add positive identify function

zhichao-li
authored and
rxin
committed
Closes #6850.
rxin
committed
[SPARK-8309] [CORE] Support for more than 12M items in OpenHashMap

SlavikBaranov
authored and
srowen
committed
[SPARK-7199] [SQL] Add date and timestamp support to UnsafeRow

viirya
authored and
Davies Liu
committed
[SPARK-6390] [SQL] [MLlib] Port MatrixUDT to PySpark

MechCoder
authored and
Davies Liu
committed
[SPARK-7017] [BUILD] [PROJECT INFRA] Refactor dev/run-tests into Python

Brennon York
authored and
JoshRosen
committed
[SPARK-8077] [SQL] Optimization for TreeNodes with large numbers of children

MickDavies
authored and
marmbrus
committed
[SPARK-8395] [DOCS] start-slave.sh docs incorrect

srowen
authored and
Andrew Or
committed
[SPARK-6782] add sbt-revolver plugin

squito
authored and
marmbrus
committed
[SPARK-8010] [SQL] Promote types to StringType as implicit conversion in non-binary expression of HiveTypeCoercion

OopsOutOfMemory
authored and
marmbrus
committed
[SPARK-8161] Set externalBlockStoreInitialized to be true, after ExternalBlockStore is initialized

Mingfei
authored and
Andrew Or
committed
[SPARK-8372] History server shows incorrect information for application not started

carsonwang
authored and
Andrew Or
committed
[SPARK-8373] [PYSPARK] Add emptyRDD to pyspark and fix the issue when calling sum on an empty RDD

zsxwing
authored and
Andrew Or
committed
[SPARK-7913] [CORE] Increase the maximum capacity of PartitionedPairBuffe, PartitionedSerializedPairBuffer and AppendOnlyMap

zsxwing
authored and
Andrew Or
committed
[SPARK-7067] [SQL] fix bug when use complex nested fields in ORDER BY

cloud-fan
authored and
marmbrus
committed
[SPARK-8306] [SQL] AddJar command needs to set the new class loader to the HiveConf inside executionHive.state.

yhuai
authored and
marmbrus
committed
[SPARK-8404] [STREAMING] [TESTS] Use thread-safe collections to make the tests more reliable

zsxwing
authored and
tdas
committed
[SPARK-8397] [SQL] Allow custom configuration for TestHive

Punya Biswal
authored and
marmbrus
committed