Python API for Spark Streaming#10

Closed

giwa wants to merge 636 commits intotdas:masterfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 5, 2014

[SPARK-2859] Update url of Kryo project in related docs

gchen
authored and
pwendell
committed
SPARK-2380: Support displaying accumulator values in the web UI
pwendell
committed
SPARK-1680: use configs for specifying environment variables on YARN
tgravescs
committed
[SPARK-2864][MLLIB] fix random seed in word2vec; move model to local
mengxr
committed
[SPARK-2503] Lower shuffle output buffer (spark.shuffle.file.buffer.kb) to 32KB.
rxin
committed
[SPARK-2550][MLLIB][APACHE SPARK] Support regularization and intercept in pyspark's linear methods

miccagiann
authored and
mengxr
committed

Commits on Aug 6, 2014

SPARK-2869 - Fix tiny bug in JdbcRdd for closing jdbc connection

Stephen Boesch
authored and
rxin
committed
[sql] rename project name in pom.xml of hive-thriftserver module

scwf
authored and
marmbrus
committed
[SPARK-2650][SQL] Try to partially fix SPARK-2650 by adjusting initial buffer size and reducing memory allocation

liancheng
authored and
marmbrus
committed
[SPARK-2854][SQL] Finalize _acceptable_types in pyspark.sql

yhuai
authored and
marmbrus
committed
[SPARK-2866][SQL] Support attributes in ORDER BY that aren't in SELECT
marmbrus
committed
[SPARK-2806] core - upgrade to json4s-jackson 3.2.10

avati
authored and
pwendell
committed
[SQL] Tighten the visibility of various SQLConf methods and renamed setter/getters

rxin
authored and
marmbrus
committed
[SQL] Fix logging warn -> debug
marmbrus
committed
SPARK-2294: fix locality inversion bug in TaskManager

CodingCat
authored and
mateiz
committed
[MLlib] Use this.type as return type in k-means' builder pattern

DB Tsai
authored and
mengxr
committed
[SPARK-1022][Streaming][HOTFIX] Fixed zookeeper dependency of Kafka

tdas
authored and
pwendell
committed
[SPARK-2157] Enable tight firewall rules for Spark

andrewor14
authored and
pwendell
committed
[SPARK-2875] [PySpark] [SQL] handle null in schemaRDD()

davies
authored and
marmbrus
committed
[SPARK-2678][Core][SQL] A workaround for SPARK-2678

liancheng
authored and
pwendell
committed
[SPARK-2627] [PySpark] have the build enforce PEP 8 automatically

nchammas
authored and
rxin
committed
SPARK-2566. Update ShuffleWriteMetrics incrementally

sryza
authored and
pwendell
committed
[SPARK-2852][MLLIB] API consistency for `mllib.feature`
mengxr
committed
[PySpark] Add blanklines to Python docstrings so example code renders correctly

rnowling
authored and
mengxr
committed
[HOTFIX][Streaming] Handle port collisions in flume polling test

andrewor14
authored and
tdas
committed
SPARK-2882: Spark build now checks local maven cache for dependencies

GregOwen
authored and
pwendell
committed

Commits on Aug 7, 2014

[SPARK-2583] ConnectionManager error reporting

sarutak
authored and
pwendell
committed
SPARK-2879 [BUILD] Use HTTPS to access Maven Central and other repos

srowen
authored and
pwendell
committed
HOTFIX: Support custom Java 7 location
pwendell
committed
WIP
giwa
committed
[SPARK-2887] fix bug of countApproxDistinct() when have more than one partition

davies
authored and
pwendell
committed
[SPARK-2851] [mllib] DecisionTree Python consistency update

jkbradley
authored and
mengxr
committed
SPARK-2879 part 2 [BUILD] Use HTTPS to access Maven Central and other repos

srowen
authored and
pwendell
committed
[mllib] DecisionTree Strategy parameter checks

jkbradley
authored and
mengxr
committed
[SPARK-2852][MLLIB] Separate model from IDF/StandardScaler algorithms
mengxr
committed
SPARK-2905 Fixed path sbin => bin

dosoft
authored and
pwendell
committed
SPARK-2899 Doc generation is back to working in new SBT Build.

ScrapCodes
authored and
pwendell
committed

Commits on Aug 8, 2014

SPARK-2787: Make sort-based shuffle write files directly when there's no sorting/aggregation and # partitions is small

mateiz
authored and
rxin
committed
SPARK-2565. Update ShuffleReadMetrics as blocks are fetched

sryza
authored and
pwendell
committed
[SPARK-2904] Remove non-used local variable in SparkSubmitArguments

sarutak
authored and
pwendell
committed
[SPARK-2911]: provide rdd.parent[T](j) to obtain jth parent RDD

erikerlandson
authored and
rxin
committed
[SPARK-2888] [SQL] Fix addColumnMetadataToConf in HiveTableScan

yhuai
authored and
marmbrus
committed
[SPARK-2908] [SQL] JsonRDD.nullTypeToStringType does not convert all NullType to StringType

yhuai
authored and
marmbrus
committed
[SPARK-2877] [SQL] MetastoreRelation should use SparkClassLoader when creating the tableDesc

yhuai
authored and
marmbrus
committed
[SPARK-2919] [SQL] Basic support for analyze command in HiveQl

yhuai
authored and
marmbrus
committed
[SPARK-2700] [SQL] Hidden files (such as .impala_insert_staging) should be filtered out by sqlContext.parquetFile

chutium
authored and
marmbrus
committed
[SPARK-1997][MLLIB] update breeze to 0.9
mengxr
committed
[SPARK-2897][SPARK-2920]TorrentBroadcast does use the serializer class specified in the spark option "spark.serializer"

witgo
authored and
rxin
committed

Commits on Aug 9, 2014

[Web UI]Make decision order of Worker's WebUI port consistent with Master's

WangTaoTheTonic
authored and
rxin
committed
[SPARK-2911] apply parent[T](j) to clarify UnionRDD code

erikerlandson
authored and
rxin
committed
[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode

li-zhihui
authored and
pwendell
committed
[SPARK-2861] Fix Doc comment of histogram method

Chandan Kumar
authored and
pwendell
committed

Commits on Aug 10, 2014

[SPARK-1766] sorted functions to meet pedantic requirements

Chris Cope
authored and
pwendell
committed
[SPARK-2894] spark-shell doesn't accept flags

sarutak
authored and
pwendell
committed
Updated Spark SQL README to include the hive-thriftserver module
rxin
committed
Turn UpdateBlockInfo into case class.
rxin
committed
Remove extra semicolon in Task.scala

witgo
authored and
rxin
committed
[SPARK-2950] Add gc time and shuffle write time to JobLogger
shivaram
committed
[SPARK-2898] [PySpark] fix bugs in deamon.py

davies
authored and
JoshRosen
committed
[SPARK-2937] Separate out samplyByKeyExact as its own API in PairRDDFunction

dorx
authored and
mengxr
committed

Commits on Aug 11, 2014

Commits on Aug 12, 2014

[SPARK-2931] In TaskSetManager, reset currentLocalityIndex after recomputing locality levels
JoshRosen
committed
[SPARK-2515][mllib] Chi Squared test

dorx
authored and
mengxr
committed
[SPARK-2934][MLlib] Adding LogisticRegressionWithLBFGS Interface

DB Tsai
authored and
mengxr
committed
[SPARK-2844][SQL] Correctly set JVM HiveContext if it is passed into Python HiveContext constructor

ahirreddy
authored and
marmbrus
committed
[SPARK-2590][SQL] Added option to handle incremental collection, disabled by default

liancheng
authored and
marmbrus
committed
[sql]use SparkSQLEnv.stop() in ShutdownHook

scwf
authored and
marmbrus
committed
[SQL] A tiny refactoring in HiveContext#analyze

yhuai
authored and
marmbrus
committed
[SPARK-2965][SQL] Fix HashOuterJoin output nullabilities.

ueshin
authored and
marmbrus
committed
[SPARK-2968][SQL] Fix nullabilities of Explode.

ueshin
authored and
marmbrus
committed
[SPARK-2650][SQL] Build column buffers in smaller batches
marmbrus
committed
[SQL] [SPARK-2826] Reduce the memory copy while building the hashmap for HashOuterJoin

chenghao-intel
authored and
marmbrus
committed
[SPARK-2923][MLLIB] Implement some basic BLAS routines
mengxr
committed
[MLlib] Correctly set vectorSize and alpha

Ishiihara
authored and
mengxr
committed
fix flaky tests

davies
authored and
marmbrus
committed

Commits on Aug 13, 2014

SPARK-2830 [MLlib]: re-organize mllib documentation

atalwalkar
authored and
mengxr
committed
[SPARK-2953] Allow using short names for io compression codecs
rxin
committed
Use transferTo when copy merge files in ExternalSorter

colorant
authored and
rxin
committed
[SPARK-1777 (partial)] bugfix: make size of requested memory correctly

liyezhang556520
authored and
rxin
committed
[SPARK-2993] [MLLib] colStats (wrapper around MultivariateStatisticalSummary) in Statistics

dorx
authored and
mengxr
committed
[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL

sarutak
authored and
marmbrus
committed
[SPARK-3013] [SQL] [PySpark] convert array into list

davies
authored and
marmbrus
committed
[SPARK-2983] [PySpark] improve performance of sortByKey()

davies
authored and
mateiz
committed
[MLLIB] use Iterator.fill instead of Array.fill
mengxr
committed
[SPARK-3004][SQL] Added null checking when retrieving row set

liancheng
authored and
marmbrus
committed
[SPARK-2817] [SQL] add "show create table" support

tianyi
authored and
marmbrus
committed

Commits on Aug 14, 2014

[SPARK-2994][SQL] Support for udfs that take complex types
marmbrus
committed
[SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer

liancheng
authored and
marmbrus
committed
[SPARK-2935][SQL]Fix parquet predicate push down bug
marmbrus
committed
[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled

sarutak
authored and
marmbrus
committed
[SPARK-2986] [SQL] fixed: setting properties does not effect

guowei
authored and
marmbrus
committed
SPARK-3020: Print completed indices rather than tasks in web UI

pwendell
authored and
rxin
committed
WIP: solved partitioned and None is not recognized
giwa
committed
[SPARK-3006] Failed to execute spark-shell in Windows OS

tsudukim
authored and
andrewor14
committed
[Docs] Add missing <code> tags (minor)

andrewor14
authored and
rxin
committed
[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel
mengxr
committed
[SPARK-3029] Disable local execution of Spark jobs by default

aarondav
authored and
rxin
committed
broke something
giwa
committed
SPARK-2893: Do not swallow Exceptions when running a custom kryo registrator

GrahamDennis
authored and
rxin
committed
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile

josephsu
authored and
marmbrus
committed
[SPARK-2927][SQL] Add a conf to configure if we always read Binary columns stored in Parquet as String columns

yhuai
authored and
marmbrus
committed
[SQL] Python JsonRDD UTF8 Encoding Fix

ahirreddy
authored and
marmbrus
committed
[SPARK-2925] [sql]fix spark-sql and start-thriftserver shell bugs when set --driver-java-options

scwf
authored and
marmbrus
committed
Minor cleanup of metrics.Source
rxin
committed
[SPARK-2979][MLlib] Improve the convergence rate by minimizing the condition number

DB Tsai
authored and
mengxr
committed
Revert [SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile
marmbrus
committed
SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...

jacek-lewandowski
authored and
andrewor14
committed
Make dev/mima runnable on Mac OS X.
rxin
committed

Commits on Aug 15, 2014

Commits on Aug 16, 2014

Commits on Aug 17, 2014

In the stop method of ConnectionManager to cancel the ackTimeoutMonitor

witgo
authored and
JoshRosen
committed
[SPARK-3077][MLLIB] fix some chisq-test
mengxr
committed
[SPARK-3042] [mllib] DecisionTree Filter top-down instead of bottom-up

jkbradley
authored and
mengxr
committed

Commits on Aug 18, 2014

SPARK-2881. Upgrade snappy-java to 1.1.1.3.
pwendell
committed
Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled"
marmbrus
committed
[SQL] Improve debug logging and toStrings.
marmbrus
committed
[SPARK-1981] updated streaming-kinesis.md

cfregly
authored and
tdas
committed
[HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests.

harishreedharan
authored and
tdas
committed
[SPARK-3087][MLLIB] fix col indexing bug in chi-square and add a check for number of distinct values
mengxr
committed
SPARK-2884: Create binary builds in parallel with release script.
pwendell
committed
SPARK-2900. aggregate inputBytes per stage

sryza
authored and
pwendell
committed
[SPARK-3097][MLlib] Word2Vec performance improvement

Ishiihara
authored and
mengxr
committed
[SPARK-2842][MLlib]Word2Vec documentation

Ishiihara
authored and
mengxr
committed
add comments
giwa
committed
removed wasted print in DStream
giwa
committed
added sparkContext as input parameter in StreamingContext
giwa
committed
added gorupByKey testcase
giwa
committed
added testcase for combineByKey
giwa
committed
initial commit for pySparkStreaming
giwa
committed
comment PythonDStream.PairwiseDStream

Ken Takagiwa
authored and
giwa
committed
modify dstream.py to fix indent error

Ken Takagiwa
authored and
giwa
committed
added reducedByKey not working yet

Ken Takagiwa
authored and
giwa
committed
implementing transform function in Python

Ken Takagiwa
authored and
giwa
committed
modified the code base on comment in https://github.com/tdas/spark/pull/10

Ken Takagiwa
authored and
giwa
committed
add coment for hack why PYSPARK_PYTHON is needed in spark-submit

Ken Takagiwa
authored and
giwa
committed
add coment for hack why PYSPARK_PYTHON is needed in spark-submit

Ken Takagiwa
authored and
giwa
committed
remove not implemented DStream functions in python

Ken Takagiwa
authored and
giwa
committed
revert pom.xml

Ken Takagiwa
authored and
giwa
committed
sorted the import following Spark coding convention

Ken Takagiwa
authored and
giwa
committed
add empty line

Ken Takagiwa
authored and
giwa
committed
remove unused import in python

Ken Takagiwa
authored and
giwa
committed
initial commit for socketTextStream

Ken Takagiwa
authored and
giwa
committed
fied input of socketTextDStream

Ken Takagiwa
authored and
giwa
committed
added doctest for pyspark.streaming.duration

Ken Takagiwa
authored and
giwa
committed
fixed typo of network_workdcount.py

Ken Takagiwa
authored and
giwa
committed
delete old file

Ken Takagiwa
authored and
giwa
committed
added reducedByKey not working yet

Ken Takagiwa
authored and
giwa
committed
reduceByKey is working

Ken Takagiwa
authored and
giwa
committed
edit python sparkstreaming example

Ken Takagiwa
authored and
giwa
committed
added count operation but this implementation need double check

Ken Takagiwa
authored and
giwa
committed
fix map function

Ken Takagiwa
authored and
giwa
committed
clean up code

Ken Takagiwa
authored and
giwa
committed
clean up codes

Ken Takagiwa
authored and
giwa
committed
remove waste file

Ken Takagiwa
authored and
giwa
committed
Implemented DStream.foreachRDD in the Python API using Py4J callback server.

tdas
authored and
giwa
committed
Added missing file

tdas
authored and
giwa
committed
Added extra line.

tdas
authored and
giwa
committed
tried to restart callback server

Ken Takagiwa
authored and
giwa
committed
Kill py4j callback server properly

Ken Takagiwa
authored and
giwa
committed
Removed the waste line
giwa
committed
implemented reduce and count function in Dstream
giwa
committed
clean up examples
giwa
committed
added stop in StreamingContext
giwa
committed
clean up dstream.py
giwa
committed
initial commit for testcase
giwa
committed
WIP
giwa
committed
update comment
giwa
committed
WIP
giwa
committed
WIP: added PythonTestInputStream
giwa
committed
WIP
giwa
committed
WIP added test case
giwa
committed
added basic operation test cases
giwa
committed
delete waste file
giwa
committed
fixed PEP-008 violation
giwa
committed
remove export PYSPARK_PYTHON in spark submit
giwa
committed
removed unnesessary changes
giwa
committed
edited the comment to add more precise description
giwa
committed
added mapValues and flatMapVaules WIP for glom and mapPartitions test
giwa
committed
WIP: solved partitioned and None is not recognized
giwa
committed
broke something
giwa
committed
all tests are passed if numSlice is 2 and the numver of each input is over 4
giwa
committed
basic function test cases are passed
giwa
committed
modified streaming test case to add coment
giwa
committed
remove waste duplicated code
giwa
committed
added saveAsTextFiles and saveAsPickledFiles
giwa
committed
added TODO coments
giwa
committed
add comments
giwa
committed
removed wasted print in DStream
giwa
committed
added sparkContext as input parameter in StreamingContext
giwa
committed
added gorupByKey testcase
giwa
committed
added testcase for combineByKey
giwa
committed
merge with master
giwa
committed

Commits on Aug 20, 2014

clean up
giwa
committed

Commits on Aug 21, 2014