Skip to content

[DOC] Minor modification to Streaming docs with regards to parallel data receiving #6544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nishkamravi2 wants to merge 69 commits into from
Closed

[DOC] Minor modification to Streaming docs with regards to parallel data receiving #6544

nishkamravi2 wants to merge 69 commits into from

Conversation

@nishkamravi2
Copy link
Contributor

@nishkamravi2 nishkamravi2 commented Jun 1, 2015

nishkamravi2 and others added 30 commits June 3, 2014 15:28
@nishkamravi2
Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeT…
681b36f 
…extFiles

The prefix "file:" is missing in the string inserted as key in HashMap
@nishkamravi2
Fix in Spark for the Concurrent thread modification issue (SPARK-1097,…
5108700 
… HADOOP-10456)
@nishkamravi2
Undo the fix for SPARK-1758 (the problem is fixed)
6b840f0
@nishkamravi2
Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
df2aeb1
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark
eb663ca
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark
5423a03
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark
3bf8fad
@nishkamravi2
Accept memory input as "30g", "512M" instead of an int value, to be c…
2b630f9 
…onsistent with rest of Spark
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark
efd688a
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
2e69f11 
…nravi
@nishkamravi2
Modify default YARN memory_overhead-- from an additive constant to a …
ebcde10 
…multiplier (redone to resolve merge conflicts)
@nishkamravi2
Update YarnAllocator.scala
1cf2d1e
@nishkamravi2
Improving logging for AM memoryOverhead
f00fa31
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
c726bd9 
…nravi

Conflicts:
	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
@nishkamravi2
Additional changes for yarn memory overhead
362da5e
@nishkamravi2
Additional changes for yarn memory overhead issue
42c2c3d
@nishkamravi2
Additional documentation for yarn memory overhead issue
dac1047
@nishkamravi2
Remove out
5ac2ec1
@nishkamravi2
Slight change in the doc for yarn memory overhead
35daa64
@nishkamravi2
Doc change for yarn memory overhead
8f76c8b
@nishkamravi2
Update YarnAllocator.scala
636a9ff
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
5f8f9ed 
…nravi

Conflicts:
	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@nishkamravi2
Changes to maxResultSize code (improve error message and add conditio…
9f6583e 
…n to check if maxResultSize > 0)
@nishkamravi2
Modify test for maxResultSize
3e1b616
@nishkamravi2
Update TaskSetManager.scala
535295a
@nishkamravi2
Update TaskSetManagerSuite.scala
5c9a4cb
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
b446edc 
…nravi
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
79ea8b4 
…nravi
@nishkamravi2
Workaround for IllegalStateException caused by recent changes to Bloc…
f0d12de 
…kManager.stop
@nishkamravi2
Update DiskBlockManager.scala
3c5ddba
nishkamravi2 and others added 20 commits March 24, 2015 16:42
@nishkamravi2
Locate the jar using getLocation, obviates the need to pass assembly …
458af39 
…path as an argument
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
4a45d6a 
…nravi

Conflicts:
	launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
b7f4ae7 
…nravi
@nishkamravi2
Minor changes and bug fixes
36a6f87
@nishkamravi2
Update Main.java
bad4349
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
1c13b79 
…nravi

Conflicts:
	bin/spark-class
	bin/spark-class2.cmd
	launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
2be1e76 
…nravi
@nishkamravi2
Add a try/catch block around Utils.removeShutdownHook
8691e0c
@nishkamravi2
Update Utils.scala
97c383e
@nishkamravi2
Update DiskBlockManager.scala
474e3bf
@nishkamravi2
Update Utils.scala
0f1abd0
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
1b81383 
…nravi
@nishkamravi2
Check for read permission before initiating copy
981afd2
@nishkamravi2
Update Client.scala
eaa13b5
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
059ce82 
…nravi

Conflicts:
	yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@nishkamravi2
In translating from deprecated spark.kryoserializer.buffer.mb to spar…
5190f79 
…k.kryoserializer.buffer use int conversion since fractions are not permissible
@nishkamravi2
Add a test for deprecated config spark.kryoserializer.buffer.mb
5261bf6
@nishkamravi2
Update SparkConfSuite.scala
171a53c
@nishkamravi2
Merge branch 'master' of https://github.com/apache/spark into master_…
9e59c71 
…nravi

Conflicts:
	core/src/main/scala/org/apache/spark/SparkConf.scala
@nishkamravi2
Slight modification to streaming docs
46e8c03
@AmplabJenkins
Copy link

AmplabJenkins commented Jun 1, 2015

Can one of the admins verify this patch?

@srowen
Copy link
Member

srowen commented Jun 1, 2015

That's fine. This does not warrant a JIRA.
(PS why do your PRs have a huge merge history? you'd normally just branch from an updated master right?)

@nishkamravi2 nishkamravi2 changed the title [SPARK-7959][DOC] Minor modification to Streaming docs with regards to parallel data receiving [DOC] Minor modification to Streaming docs with regards to parallel data receiving Jun 1, 2015
@nishkamravi2
Copy link
Contributor Author

nishkamravi2 commented Jun 1, 2015

Yeah, the JIRA in the title seems unnecessary (removed). With regards to the merge history: not sure, will have to look into it.

@tdas
Copy link
Contributor

tdas commented Jun 1, 2015

LGTM other than the commit history.
Can you just just checkout the spark github master branch and cherry-pick your change (commit 46e8c03) on it. That should keep the commit history clean.

@srowen
Copy link
Member

srowen commented Jun 1, 2015

FWIW I keep master in my fork up to date with git pull upstream master and git push origin master. Then whenever you make a branch, just branch from your master. Have you been merging into your master? It should just track the upstream one directly with no variance.

asfgit pushed a commit that referenced this pull request Jun 1, 2015
@nishkamravi2 @srowen
[DOC] Minor modification to Streaming docs with regards to parallel d…
2f41cf3 
…ata receiving

pwendell tdas

Author: Nishkam Ravi <[email protected]>
Author: nishkamravi2 <[email protected]>
Author: nravi <[email protected]>

Closes #6544 from nishkamravi2/master_nravi and squashes the following commits:

46e8c03 [Nishkam Ravi] Slight modification to streaming docs

(cherry picked from commit e7c7e51)
Signed-off-by: Sean Owen <[email protected]>
@asfgit asfgit closed this in e7c7e51 Jun 1, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@nishkamravi2 @jeanlyn
[DOC] Minor modification to Streaming docs with regards to parallel d…
3e0877a 
…ata receiving

pwendell tdas

Author: Nishkam Ravi <[email protected]>
Author: nishkamravi2 <[email protected]>
Author: nravi <[email protected]>

Closes apache#6544 from nishkamravi2/master_nravi and squashes the following commits:

46e8c03 [Nishkam Ravi] Slight modification to streaming docs
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nishkamravi2 @nemccarthy
[DOC] Minor modification to Streaming docs with regards to parallel d…
ded97e6 
…ata receiving

pwendell tdas

Author: Nishkam Ravi <[email protected]>
Author: nishkamravi2 <[email protected]>
Author: nravi <[email protected]>

Closes apache#6544 from nishkamravi2/master_nravi and squashes the following commits:

46e8c03 [Nishkam Ravi] Slight modification to streaming docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nishkamravi2 @AmplabJenkins @srowen @tdas