Skip to content

[SPARK-9967][SPARK-10099][STREAMING] Renamed conf spark.streaming.backpressure.{enable-->enabled} and fixed deprecated annotations #8299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tdas wants to merge 1 commit into from
Closed

[SPARK-9967][SPARK-10099][STREAMING] Renamed conf spark.streaming.backpressure.{enable-->enabled} and fixed deprecated annotations #8299

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends Serializable {
* Return an array that contains all of the elements in this RDD.
* @deprecated As of Spark 1.0.0, toArray() is deprecated, use {@link #collect()} instead
*/
@Deprecated
@deprecated("use collect()", "1.0.0")
def toArray(): JList[T] = collect()

/**
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ class SparkHadoopUtil extends Logging {
}
}

@Deprecated
@deprecated("use newConfiguration with SparkConf argument", "1.2.0")
def newConfiguration(): Configuration = newConfiguration(null)

/**
Expand Down
4 changes: 2 additions & 2 deletions streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ trait JavaDStreamLike[T, This <: JavaDStreamLike[T, This, R], R <: JavaRDDLike[T
*
* @deprecated As of release 0.9.0, replaced by foreachRDD
*/
@Deprecated
@deprecated("Use foreachRDD", "0.9.0")
def foreach(foreachFunc: JFunction[R, Void]) {
foreachRDD(foreachFunc)
}
Expand All @@ -302,7 +302,7 @@ trait JavaDStreamLike[T, This <: JavaDStreamLike[T, This, R], R <: JavaRDDLike[T
*
* @deprecated As of release 0.9.0, replaced by foreachRDD
*/
@Deprecated
@deprecated("Use foreachRDD", "0.9.0")
def foreach(foreachFunc: JFunction2[R, Time, Void]) {
foreachRDD(foreachFunc)
}
Expand Down
8 changes: 4 additions & 4 deletions streaming/src/main/scala/org/apache/spark/streaming/scheduler/RateController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,15 +76,15 @@ private[streaming] abstract class RateController(val streamUID: Int, rateEstimat
val elements = batchCompleted.batchInfo.streamIdToInputInfo

for {
processingEnd <- batchCompleted.batchInfo.processingEndTime;
workDelay <- batchCompleted.batchInfo.processingDelay;
waitDelay <- batchCompleted.batchInfo.schedulingDelay;
processingEnd <- batchCompleted.batchInfo.processingEndTime
Copy link
Contributor Author

@tdas tdas Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The semicolons where unnecessary and was being flagged by IntelliJ

workDelay <- batchCompleted.batchInfo.processingDelay
waitDelay <- batchCompleted.batchInfo.schedulingDelay
elems <- elements.get(streamUID).map(_.numRecords)
} computeAndPublish(processingEnd, elems, workDelay, waitDelay)
}
}

object RateController {
def isBackPressureEnabled(conf: SparkConf): Boolean =
conf.getBoolean("spark.streaming.backpressure.enable", false)
conf.getBoolean("spark.streaming.backpressure.enabled", false)
Copy link
Member

@srowen srowen Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here was that this flag hadn't been 'released' yet so OK to change the name without keeping the old one?

The two changes in the PR were pretty unrelated right? still I can't imagine someone would really care about back-porting one and not be able to consume the other as they're minor.

Copy link
Contributor Author

@tdas tdas Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, for both. That's why went ahead with squashing both into one.

}