User defined datatypes#13

Closed

marmbrus wants to merge 297 commits intomarmbrus:masterfrom

jkbradley:sql-udt

Commits on Oct 13, 2014

[SPARK-3861][SQL] Avoid rebuilding hash tables for broadcast joins on each partition

rxin
authored and
marmbrus
committed
[Spark] RDD take() method: overestimate too much

yingjieMiao
authored and
rxin
committed
[SPARK-3892][SQL] remove redundant type name

adrian-wang
authored and
marmbrus
committed
[SPARK-3407][SQL]Add Date type support

adrian-wang
authored and
marmbrus
committed
[SPARK-2066][SQL] Adds checks for non-aggregate attributes with aggregation

liancheng
authored and
marmbrus
committed
[SPARK-3529] [SQL] Delete the temp files after test exit

chenghao-intel
authored and
marmbrus
committed
[SPARK-3771][SQL] AppendingParquetOutputFormat should use reflection to prevent from breaking binary-compatibility.

ueshin
authored and
marmbrus
committed
[SPARK-3559][SQL] Remove unnecessary columns from List of needed Column Ids in Hive Conf

gvramana
authored and
marmbrus
committed
[SQL] Add type checking debugging functions
marmbrus
committed
SPARK-3807: SparkSql does not work for tables created using custom serde

chiragaggarwal
authored and
marmbrus
committed
[SQL]Small bug in unresolved.scala

Ishiihara
authored and
marmbrus
committed
[SPARK-3809][SQL] Fixes test suites in hive-thriftserver

liancheng
authored and
marmbrus
committed

Commits on Oct 14, 2014

[SPARK-3912][Streaming] Fixed flakyFlumeStreamSuite
tdas
committed
[SPARK-3921] Fix CoarseGrainedExecutorBackend's arguments for Standalone mode

aarondav
authored and
andrewor14
committed
SPARK-3178 setting SPARK_WORKER_MEMORY to a value without a label (m or g) sets the worker memory limit to zero

bbejeck
authored and
JoshRosen
committed
[SPARK-3946] gitignore in /python includes wrong directory

tsudukim
authored and
andrewor14
committed
[SPARK-3944][Core] Using Option[String] where value of String can be null

Shiti
authored and
andrewor14
committed
SPARK-3803 [MLLIB] ArrayIndexOutOfBoundsException found in executing computePrincipalComponents

srowen
authored and
mengxr
committed
[SPARK-3869] ./bin/spark-class miss Java version with _JAVA_OPTIONS set

cocoatomo
authored and
andrewor14
committed

Commits on Oct 15, 2014

[SPARK-3943] Some scripts bin\*.cmd pollutes environment variables in Windows

tsudukim
authored and
andrewor14
committed
SPARK-1307 [DOCS] Don't use term 'standalone' to refer to a Spark Application

srowen
authored and
mengxr
committed
[SPARK-2098] All Spark processes should support spark-defaults.conf, config file

witgo
authored and
andrewor14
committed

Commits on Oct 16, 2014

Commits on Oct 17, 2014

SPARK-3874: Provide stable TaskContext API

ScrapCodes
authored and
pwendell
committed
[SPARK-3923] Increase Akka heartbeat pause above heartbeat interval

aarondav
authored and
andrewor14
committed
[SQL]typo in HiveFromSpark

Kun Li
authored and
andrewor14
committed
[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo

liyezhang556520
authored and
andrewor14
committed
[SPARK-3890][Docs]remove redundant spark.executor.memory in doc

WangTaoTheTonic
authored and
andrewor14
committed
[SPARK-3741] Add afterExecute for handleConnectExecutor

zsxwing
authored and
JoshRosen
committed
[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes

YanTangZhai
authored and
andrewor14
committed
[SPARK-3973] Print call site information for broadcasts

shivaram
authored and
JoshRosen
committed
[SPARK-3935][Core] log the number of records that has been written

likun
authored and
andrewor14
committed
[SPARK-3979] [yarn] Use fs's default replication.

Marcelo Vanzin
authored and
tgravescs
committed
[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations

marmbrus
authored and
pwendell
committed
[SPARK-3985] [Examples] fix file path using os.path.join

adrian-wang
authored and
JoshRosen
committed
[SPARK-3934] [SPARK-3918] [mllib] Bug fixes for RandomForest, DecisionTree

jkbradley
authored and
mengxr
committed

Commits on Oct 18, 2014

SPARK-3926 [CORE] Result of JavaRDD.collectAsMap() is not Serializable

srowen
authored and
JoshRosen
committed

Commits on Oct 19, 2014

Commits on Oct 20, 2014

[SPARK-3902] [SPARK-3590] Stabilize AsynRDDActions and add Java API
JoshRosen
committed
[SPARK-3948][Shuffle]Fix stream corruption bug in sort-based shuffle

jerryshao
authored and
JoshRosen
committed
[SPARK-4010][Web UI]Spark UI returns 500 in yarn-client mode

witgo
authored and
andrewor14
committed
[SPARK-3986][SQL] Fix package names to fit their directory names.

ueshin
authored and
marmbrus
committed
[SPARK-3736] Workers reconnect when disassociated from the master.

mccheah
authored and
JoshRosen
committed
[SPARK-3207][MLLIB]Choose splits for continuous features in DecisionTree more adaptively

chouqin
authored and
mengxr
committed
[SPARK-3906][SQL] Adds multiple join support for SQLContext

liancheng
authored and
marmbrus
committed
[SPARK-3800][SQL] Clean aliases from grouping expressions
marmbrus
committed
[SPARK-3966][SQL] Fix nullabilities of Cast related to DateType.

ueshin
authored and
marmbrus
committed
[SPARK-3945]Properties of hive-site.xml is invalid in running the Thrift JDBC server

luogankun
authored and
marmbrus
committed

Commits on Oct 21, 2014

[SPARK-3969][SQL] Optimizer should have a super class as an interface.

ueshin
authored and
marmbrus
committed
[SPARK-3940][SQL] Avoid console printing error messages three times

wangxiaojing
authored and
marmbrus
committed
Update Building Spark link.
rxin
committed
[SPARK-3958] TorrentBroadcast cleanup / debugging improvements.
JoshRosen
committed
[SPARK-4023] [MLlib] [PySpark] convert rdd into RDD of Vector

Davies Liu
authored and
mengxr
committed
replace awaitTransformation with awaitTermination in scaladoc/javadoc

holdenk
authored and
JoshRosen
committed
[SPARK-4035] Fix a wrong format specifier

zsxwing
authored and
JoshRosen
committed
[SPARK-4020] Do not rely on timeouts to remove failed block managers

andrewor14
authored and
kayousterhout
committed
SPARK-3770: Make userFeatures accessible from python

Michelangelo D'Agostino
authored and
mengxr
committed
[SPARK-3994] Use standard Aggregator code path for countByKey and countByValue

aarondav
authored and
andrewor14
committed
SPARK-3568 [mllib] add ranking metrics

coderxiang
authored and
mengxr
committed
[SQL]redundant methods for broadcast

scwf
authored and
JoshRosen
committed

Commits on Oct 22, 2014

SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy

sryza
authored and
pwendell
committed
Update JavaCustomReceiver.java

Karthik
authored and
JoshRosen
committed
use isRunningLocally rather than runningLocally

CrazyJvm
authored and
rxin
committed
Fix for sampling error in NumPy v1.9 [SPARK-3995][PYSPARK]

freeman-lab
authored and
mengxr
committed
[SPARK-3426] Fix sort-based shuffle error when spark.shuffle.compress and spark.shuffle.spill.compress settings are different

JoshRosen
authored and
andrewor14
committed
[SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1

zsxwing
authored and
andrewor14
committed

Commits on Oct 23, 2014

[SPARK-3812] [BUILD] Adapt maven build to publish effective pom.

ScrapCodes
authored and
pwendell
committed
[BUILD] Fixed resolver for scalastyle plugin and upgrade sbt version.

ScrapCodes
authored and
pwendell
committed
[SPARK-4055][MLlib] Inconsistent spelling 'MLlib' and 'MLLib'

sarutak
authored and
mengxr
committed
[SPARK-4006] In long running contexts, we encountered the situation of double registe...

tsliwowicz
authored and
andrewor14
committed
specify unidocGenjavadocVersion of 0.8

holdenk
authored and
mengxr
committed
Revert "[SPARK-3812] [BUILD] Adapt maven build to publish effective pom."
pwendell
committed
[SPARK-4019] [SPARK-3740] Fix MapStatus compression bug that could lead to empty results or Snappy errors

JoshRosen
authored and
pwendell
committed

Commits on Oct 24, 2014

[SPARK-3993] [PySpark] fix bug while reuse worker after take()

davies
authored and
JoshRosen
committed
[SPARK-4000][BUILD] Sends archived unit tests logs to Jenkins master

liancheng
authored and
JoshRosen
committed
SPARK-3812 Build changes to publish effective pom.

ScrapCodes
authored and
pwendell
committed
[SPARK-2652] [PySpark] donot use KyroSerializer as default serializer

Davies Liu
authored and
mengxr
committed
[SPARK-3900][YARN] ApplicationMaster's shutdown hook fails and IllegalStateException is thrown.

sarutak
authored and
tgravescs
committed
[SPARK-4051] [SQL] [PySpark] Convert Row into dictionary

Davies Liu
authored and
JoshRosen
committed
[SPARK-4050][SQL] Fix caching of temporary tables with projections.
marmbrus
committed
[SPARK-2706][SQL] Enable Spark to support Hive 0.13

zhzhan
authored and
marmbrus
committed
[SPARK-4026][Streaming] Write ahead log management

harishreedharan
authored and
tdas
committed
[SPARK-2713] Executors of same application in same host should only download files & jars once

li-zhihui
authored and
Andrew Or
committed
[SPARK-4076] Parameter expansion in spark-config is wrong

sarutak
authored and
Andrew Or
committed
[SPARK-4075] [Deploy] Jar url validation is not enough for Jar file

sarutak
authored and
Andrew Or
committed
[SPARK-4013] Do not create multiple actor systems on each executor

andrewor14
authored and
Andrew Or
committed
[SPARK-4067] refactor ExecutorUncaughtExceptionHandler

Nan Zhu
authored and
Andrew Or
committed
[GraphX] Modify option name according to example doc in SynthBenchmark

GraceH
authored and
ankurdave
committed
[HOTFIX][SQL] Remove sleep on reset() failure.

marmbrus
authored and
JoshRosen
committed
[SPARK-4080] Only throw IOException from [write|read][Object|External]
JoshRosen
committed

Commits on Oct 25, 2014

[SPARK-4056] Upgrade snappy-java to 1.1.1.5
JoshRosen
committed
[SQL] Update Hive test harness for Hive 12 and 13

marmbrus
authored and
JoshRosen
committed
[SPARK-2321] Stable pull-based progress / status API
JoshRosen
committed
[SPARK-4088] [PySpark] Python worker should exit after socket is closed by JVM

Davies Liu
authored and
JoshRosen
committed

Commits on Oct 26, 2014

Revert "[SPARK-4056] Upgrade snappy-java to 1.1.1.5"
JoshRosen
committed
[SPARK-4071] Unroll fails silently if BlockManager is small

Andrew Or
authored and
JoshRosen
committed
SPARK-3359 [DOCS] sbt/sbt unidoc doesn't work with Java 8

srowen
authored and
mengxr
committed
Update RoaringBitmap to 0.4.3

lemire
authored and
JoshRosen
committed
[SPARK-3616] Add basic Selenium tests to WebUISuite
JoshRosen
committed
Just fixing comment that shows usage

AtlasPilotPuppy
authored and
Andrew Or
committed
[SPARK-3925][SQL] Do not consider the ordering of qualifiers during comparison

viirya
authored and
marmbrus
committed
[HOTFIX][SQL] Temporarily turn off hive-server tests.
marmbrus
committed
[SPARK-3537][SPARK-3914][SQL] Refines in-memory columnar table statistics

liancheng
authored and
marmbrus
committed
[SPARK-3997][Build]scalastyle should output the error location

witgo
authored and
JoshRosen
committed
[SQL][DOC] Wrong package name "scala.math.sql" in sql-programming-guide.md

sarutak
authored and
marmbrus
committed
[SPARK-3953][SQL][Minor] Confusable variable name.

sarutak
authored and
marmbrus
committed
[SPARK-4052][SQL] Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

yhuai
authored and
marmbrus
committed
[SPARK-4068][SQL] NPE in jsonRDD schema inference

yhuai
authored and
marmbrus
committed
[SPARK-3483][SQL] Special chars in column names

ravipesala
authored and
marmbrus
committed
[SPARK-3959][SPARK-3960][SQL] SqlParser fails to parse literal -9223372036854775808 (Long.MinValue). / We can apply unary minus only to literal.

sarutak
authored and
marmbrus
committed
[SPARK-4061][SQL] We cannot use EOL character in the operand of LIKE predicate.

sarutak
authored and
marmbrus
committed
[SPARK-4042][SQL] Append columns ids and names before broadcast

scwf
authored and
marmbrus
committed

Commits on Oct 27, 2014

[SPARK-3970] Remove duplicate removal of local dirs

viirya
authored and
Andrew Or
committed
[SPARK-4030] Make destroy public for broadcast variables
shivaram
committed
[SPARK-4032] Deprecate YARN alpha support in Spark 1.2

ScrapCodes
authored and
pwendell
committed
SPARK-2621. Update task InputMetrics incrementally

sryza
authored and
pwendell
committed
[SQL] Fixes caching related JoinSuite failure

liancheng
authored and
marmbrus
committed
SPARK-4022 [CORE] [MLLIB] Replace colt dependency (LGPL) with commons-math

srowen
authored and
mengxr
committed

Commits on Oct 28, 2014

[MLlib] SPARK-3987: add test case on objective value for NNLS

coderxiang
authored and
mengxr
committed
[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding

chenghao-intel
authored and
marmbrus
committed
[SPARK-3816][SQL] Add table properties from storage handler to output jobConf

alexoss68
authored and
marmbrus
committed
[SPARK-4041][SQL] Attributes names in table scan should converted to lowercase when compare with relation attributes

scwf
authored and
marmbrus
committed
[SQL] Correct a variable name in JavaApplySchemaSuite.applySchemaToJSON

yhuai
authored and
marmbrus
committed
[SPARK-3907][SQL] Add truncate table support

wangxiaojing
authored and
marmbrus
committed
[SPARK-4064]NioBlockTransferService.fetchBlocks may cause spark to hang.

witgo
authored and
rxin
committed
fix broken links in README.md

ryan-williams
authored and
rxin
committed
[SPARK-4107] Fix incorrect handling of read() and skip() return values

JoshRosen
authored and
rxin
committed
[SPARK-3961] [MLlib] [PySpark] Python API for mllib.feature

Davies Liu
authored and
mengxr
committed
[SPARK-4116][YARN]Delete the abandoned log4j-spark-container.properties

WangTaoTheTonic
authored and
tgravescs
committed
[SPARK-4095][YARN][Minor]extract val isLaunchingDriver in ClientBase

WangTaoTheTonic
authored and
tgravescs
committed
[SPARK-4098][YARN]use appUIAddress instead of appUIHostPort in yarn-client mode

WangTaoTheTonic
authored and
tgravescs
committed
[SPARK-4031] Make torrent broadcast read blocks on use.
shivaram
committed
[SPARK-4110] Wrong comments about default settings in spark-daemon.sh

sarutak
authored and
Andrew Or
committed
[SPARK-4096][YARN]let ApplicationMaster accept executor memory argument in same format as JVM memory strings

WangTaoTheTonic
authored and
Andrew Or
committed
[SPARK-3657] yarn alpha YarnRMClientImpl throws NPE appMasterRequest.setTrackingUrl starting spark-shell

sarutak
authored and
Andrew Or
committed
[SPARK-4089][Doc][Minor] The version number of Spark in _config.yaml is wrong.

sarutak
authored and
Andrew Or
committed
[SPARK-4065] Add check for IPython on Windows

msjgriffiths
authored and
Andrew Or
committed
[SPARK-4058] [PySpark] Log file name is hard coded even though there is a variable '$LOG_FILE '

sarutak
authored and
Andrew Or
committed
[SPARK-3814][SQL] Support for Bitwise AND(&), OR(|) ,XOR(^), NOT(~) in Spark HQL and SQL

ravipesala
authored and
marmbrus
committed
[SPARK-3988][SQL] add public API for date type

adrian-wang
authored and
marmbrus
committed
[Spark 3922] Refactor spark-core to use Utils.UTF_8

zsxwing
authored and
rxin
committed
[SPARK-3343] [SQL] Add serde support for CTAS

chenghao-intel
authored and
marmbrus
committed
[SPARK-4084] Reuse sort key in Sorter

mengxr
authored and
aarondav
committed

Commits on Oct 29, 2014

[SPARK-4008] Fix "kryo with fold" in KryoSerializerSuite

zsxwing
authored and
aarondav
committed
[SPARK-3904] [SQL] add constant objectinspector support for udfs

chenghao-intel
authored and
marmbrus
committed
[SPARK-4133] [SQL] [PySpark] type conversionfor python udf

Davies Liu
authored and
marmbrus
committed
[FIX] disable benchmark code
mengxr
committed
[SPARK-4129][MLlib] Performance tuning in MultivariateOnlineSummarizer

DB Tsai
authored and
mengxr
committed
[SPARK-3453] Netty-based BlockTransferService, extracted from Spark core
rxin
committed
[SPARK-4003] [SQL] add 3 types for java SQL context

adrian-wang
authored and
marmbrus
committed
[SPARK-3822] Executor scaling mechanism for Yarn
Andrew Or
committed
[SPARK-4097] Fix the race condition of 'thread'

zsxwing
authored and
rxin
committed

Commits on Oct 30, 2014

[SPARK-3795] Heuristics for dynamically scaling executors

andrewor14
authored and
Andrew Or
committed
[SPARK-4053][Streaming] Made the ReceiverSuite test more reliable, by fixing block generator throttling

tdas
authored and
Andrew Or
committed
[SPARK-1720][SPARK-1719] use LD_LIBRARY_PATH instead of -Djava.library.path

witgo
authored and
Andrew Or
committed
[SPARK-4102] Remove unused ShuffleReader.stop() method.
kayousterhout
committed
[SPARK-4130][MLlib] Fixing libSVM parser bug with extra whitespace

jegonzal
authored and
mengxr
committed
SPARK-4111 [MLlib] add regression metrics

Yanbo Liang
authored and
mengxr
committed
[SPARK-4028][Streaming] ReceivedBlockHandler interface to abstract the functionality of storage of received data
tdas
committed
[SPARK-4027][Streaming] WriteAheadLogBackedBlockRDD to read received either from BlockManager or WAL in HDFS
tdas
committed
[SPARK-4078] New FsPermission instance w/o FsPermission.createImmutable in eventlog

GraceH
authored and
Andrew Or
committed
[SPARK-3319] [SPARK-3338] Resolve Spark submit config paths

andrewor14
authored and
Andrew Or
committed
[SPARK-4138][SPARK-4139] Improve dynamic allocation settings
Andrew Or
committed
[Minor] A few typos in comments and log messages

andrewor14
authored and
Andrew Or
committed
[SPARK-4155] Consolidate usages of <driver>
Andrew Or
committed
Minor style hot fix after #2711
Andrew Or
committed
[SPARK-4153][WebUI] Update the sort keys for HistoryPage

zsxwing
authored and
Andrew Or
committed
[SPARK-3661] Respect spark.*.memory in cluster mode
Andrew Or
committed
SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce}Util should not use package org.apache.hadoop

srowen
authored and
Andrew Or
committed

Commits on Oct 31, 2014

[SPARK-4120][SQL] Join of multiple tables with syntax like SELECT .. FROM T1,T2,T3.. does not work in SparkSQL

ravipesala
authored and
marmbrus
committed
[SPARK-3968][SQL] Use parquet-mr filter2 api

Yash Datta
authored and
marmbrus
committed
Revert "SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce}Util should not use package org.apache.hadoop"
Andrew Or
committed
HOTFIX: Clean up build in network module.

pwendell
authored and
aarondav
committed
[SPARK-4124] [MLlib] [PySpark] simplify serialization in MLlib Python API

Davies Liu
authored and
mengxr
committed
[SPARK-3250] Implement Gap Sampling optimization for random sampling

erikerlandson
authored and
mengxr
committed
[SPARK-4108][SQL] Fixed usage of deprecated in sql/catalyst/types/datatypes

AtlasPilotPuppy
authored and
marmbrus
committed
[SPARK-4143] [SQL] Move inner class DeferredObjectAdapter to top level

chenghao-intel
authored and
marmbrus
committed
SPARK-3837. Warn when YARN kills containers for exceeding memory limits

sryza
authored and
tgravescs
committed
[SPARK-4016] Allow user to show/hide UI metrics.

kayousterhout
authored and
JoshRosen
committed
[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1

scwf
authored and
marmbrus
committed
[SPARK-4077][SQL] Spark SQL return wrong values for valid string timestamp values

gvramana
authored and
marmbrus
committed
[SPARK-4154][SQL] Query does not work if it has "not between " in Spark SQL and HQL

ravipesala
authored and
marmbrus
committed
[SPARK-2220][SQL] Fixes remaining Hive commands

liancheng
authored and
marmbrus
committed
[SPARK-4141] Hide Accumulators column on stage page when no accumulators exist

mmm
authored and
JoshRosen
committed
[SPARK-4150][PySpark] return self in rdd.setName

mengxr
authored and
JoshRosen
committed
[SPARK-3870] EOL character enforcement

sarutak
authored and
JoshRosen
committed
[HOT FIX] Yarn stable tests don't compile
andrewor14
committed

Commits on Nov 1, 2014

SPARK-4175. Exception on stage page

sryza
authored and
kayousterhout
committed
[MLLIB] SPARK-2329 Add multi-label evaluation metrics

avulanov
authored and
mengxr
committed
[SPARK-3838][examples][mllib][python] Word2Vec example in python

AtlasPilotPuppy
authored and
mengxr
committed
[MLLIB] SPARK-1547: Add Gradient Boosting to MLlib

manishamde
authored and
mengxr
committed
Streaming KMeans [MLLIB][SPARK-3254]

freeman-lab
authored and
mengxr
committed
Upgrading to roaring 0.4.5 (bug fix release)

lemire
authored and
rxin
committed
[SPARK-4142][GraphX] Default numEdgePartitions

jegonzal
authored and
rxin
committed
[SPARK-4115][GraphX] Add overrided count for edge counting of EdgeRDD.

luluorta
authored and
rxin
committed
[SPARK-2759][CORE] Generic Binary File Support in Spark

kmader
authored and
mateiz
committed
[SPARK-4183] Enable NettyBlockTransferService by default

aarondav
authored and
pwendell
committed
[SPARK-3569][SQL] Add metadata field to StructField

mengxr
authored and
marmbrus
committed
[SPARK-3796] Create external service which can serve shuffle files

aarondav
authored and
rxin
committed
[SPARK-4037][SQL] Removes the SessionState instance created in HiveThriftServer2

liancheng
authored and
marmbrus
committed
Revert "[SPARK-4183] Enable NettyBlockTransferService by default"
pwendell
committed
[SPARK-4121] Set commons-math3 version based on hadoop profiles, instead of shading

mengxr
authored and
pwendell
committed
[SPARK-3161][MLLIB] Adding a node Id caching mechanism for training deci...

Sung Chung
authored and
mengxr
committed

Commits on Nov 2, 2014