[SPARK-45685][CORE][SQL] Use LazyList instead of Stream

[LuciferYang]

What changes were proposed in this pull request?

This pr change to use LazyList instead of Stream due to Stream has been marked as deprecated after Scala 2.13.0.

• class Stream

@deprecated("Use LazyList (which is fully lazy) instead of Stream (which has a lazy tail only)", "2.13.0")
@SerialVersionUID(3L)
sealed abstract class Stream[+A] extends AbstractSeq[A]
  with LinearSeq[A]
  with LinearSeqOps[A, Stream, Stream[A]]
  with IterableFactoryDefaults[A, Stream]
  with Serializable {
  ...
  @deprecated("The `append` operation has been renamed `lazyAppendedAll`", "2.13.0")
  @inline final def append[B >: A](rest: => IterableOnce[B]): Stream[B] = lazyAppendedAll(rest)

• object Stream

@deprecated("Use LazyList (which is fully lazy) instead of Stream (which has a lazy tail only)", "2.13.0")
@SerialVersionUID(3L)
object Stream extends SeqFactory[Stream] {

• type Stream and value Stream

@deprecated("Use LazyList instead of Stream", "2.13.0")
type Stream[+A] = scala.collection.immutable.Stream[A]
@deprecated("Use LazyList instead of Stream", "2.13.0")
val Stream = scala.collection.immutable.Stream

• method toStream in trait IterableOnceOps

@deprecated("Use .to(LazyList) instead of .toStream", "2.13.0")
@inline final def toStream: immutable.Stream[A] = to(immutable.Stream)

Why are the changes needed?

Clean up deprecated Scala API usage.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Acitons

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

Test first, will update pr description later

[dongjoon-hyun]

+1, LGTM. Thank you, @LuciferYang .
Merged to master for Apache Spark 4.0.0.

[LuciferYang]

Thanks @dongjoon-hyun

[beliefer]

LGTM later.

[LuciferYang]

Thanks @beliefer ~

[JoshRosen]

I think that this PR may have introduced a rare performance / StackOverflowError regression:

Even though Stream is deprecated in 2.13, it is not removed and thus is is possible that some parts of Spark / Catalyst (or third-party code) might continue to pass around Stream instances, but at call sites like

val inputVars = inputVarsCandidate match {
  case stream: Stream[ExprCode] => stream.force
  case other => other
}

this PR has replaced the pattern matching with

val inputVars = inputVarsCandidate match {
  case stream: LazyList[ExprCode] => stream.force
  case other => other
}

instead of handling both Stream and LazyList, as in

val inputVars = inputVarsCandidate match {
  case stream: Stream[ExprCode] => stream.force
  case stream: LazyList[ExprCode] => stream.force
  case other => other
}

Given this, I think that we should make a followup patch to update all of the modified .force call sites to perform the forcing for both LazyList and Stream, since otherwise we would be losing the eager materialization for Streams that happen to flow to these call sites.

@LuciferYang @dongjoon-hyun , WDYT?

[LuciferYang]

@JoshRosen I have submitted a pr: #46970, please let me know if I have correctly understood your suggestion.

If current pr has a significant impact on performance, we can also revert it.

[LuciferYang]

I think that this PR may have introduced a rare performance / StackOverflowError regression:

I would like to understand more, is this issue due to the design of LazyList itself? Should we submit an issue to the Scala community?

Sorry, please ignore this question, I should have understood what you mean.