Skip to content

[SPARK-12525] Fix fatal compiler warnings in Kinesis ASL due to @transient annotations #10479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 5 commits into from
Closed

[SPARK-12525] Fix fatal compiler warnings in Kinesis ASL due to @transient annotations #10479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0b79c39
Fix compiler warnings due to @transient annotations.
JoshRosen Dec 25, 2015
70e7032
Update KinesisBackedBlockRDD.scala
JoshRosen Dec 26, 2015
afe11f2
Update KinesisBackedBlockRDD.scala
JoshRosen Dec 26, 2015
0a3f671
Update KinesisBackedBlockRDD.scala
JoshRosen Dec 26, 2015
422ef49
Rename blockIds to _blockIds
JoshRosen Dec 28, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions ...kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,26 +70,26 @@ class KinesisBackedBlockRDDPartition(
*/
private[kinesis]
class KinesisBackedBlockRDD[T: ClassTag](
@transient sc: SparkContext,
sc: SparkContext,
Copy link
Contributor Author

@JoshRosen JoshRosen Dec 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the change made to c1bc4f4#diff-3fdb38876be7c77f4568be49edba4c02L38

val regionName: String,
val endpointUrl: String,
@transient blockIds: Array[BlockId],
@transient private val _blockIds: Array[BlockId],
@transient val arrayOfseqNumberRanges: Array[SequenceNumberRanges],
@transient isBlockIdValid: Array[Boolean] = Array.empty,
@transient private val isBlockIdValid: Array[Boolean] = Array.empty,
val retryTimeoutMs: Int = 10000,
val messageHandler: Record => T = KinesisUtils.defaultMessageHandler _,
val awsCredentialsOption: Option[SerializableAWSCredentials] = None
) extends BlockRDD[T](sc, blockIds) {
) extends BlockRDD[T](sc, _blockIds) {

require(blockIds.length == arrayOfseqNumberRanges.length,
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")

override def isValid(): Boolean = true

override def getPartitions: Array[Partition] = {
Array.tabulate(blockIds.length) { i =>
Array.tabulate(_blockIds.length) { i =>
val isValid = if (isBlockIdValid.length == 0) true else isBlockIdValid(i)
new KinesisBackedBlockRDDPartition(i, blockIds(i), isValid, arrayOfseqNumberRanges(i))
new KinesisBackedBlockRDDPartition(i, _blockIds(i), isValid, arrayOfseqNumberRanges(i))
}
}

Expand Down
2 changes: 1 addition & 1 deletion ...s/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import org.apache.spark.streaming.scheduler.ReceivedBlockInfo
import org.apache.spark.streaming.{Duration, StreamingContext, Time}

private[kinesis] class KinesisInputDStream[T: ClassTag](
@transient _ssc: StreamingContext,
_ssc: StreamingContext,
Copy link
Contributor Author

@JoshRosen JoshRosen Dec 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this matches the change that was made to FlumePollingInputDStream: c1bc4f4#diff-0a2347be859b78b562e3e8e8e17a1f1fL49

streamName: String,
endpointUrl: String,
regionName: String,
Expand Down