[SPARK-40660][CORE][SQL] Switch to XORShiftRandom to distribute elements

[wangyum]

What changes were proposed in this pull request?

This PR replaces Random(hashing.byteswap32(index)) with XORShiftRandom(index) to distribute elements evenly across output partitions.

Why are the changes needed?

It seems that the distribution using XORShiftRandom is better. For example:

1. The number of output files has changed since SPARK-40407. Some downstream projects use repartition to determine the number of output files in the test.

   bin/spark-shell --master "local[2]"
   spark.range(10).repartition(10).write.mode("overwrite").parquet("/tmp/spark/repartition")
   

   Before this PR and after SPARK-40407, the number of output files is 8. After this PR or before SPARK-40407, the number of output files is 10.

2. The distribution using XORShiftRandom seem better.

   import java.util.Random
   import org.apache.spark.util.random.XORShiftRandom
   import scala.util.hashing
   
   def distribution(count: Int, partition: Int) = {
     println((1 to count).map(partitionId => new Random(partitionId).nextInt(partition))
       .groupBy(f => f)
       .map(_._2.size).mkString(". "))
   
     println((1 to count).map(partitionId => new Random(hashing.byteswap32(partitionId)).nextInt(partition))
       .groupBy(f => f)
       .map(_._2.size).mkString(". "))
   
     println((1 to count).map(partitionId => new XORShiftRandom(partitionId).nextInt(partition))
       .groupBy(f => f)
       .map(_._2.size).mkString(". "))
   }
   
   distribution(200, 4)

   The output:

   200
   50. 60. 46. 44
   55. 48. 43. 54
   

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

[wangyum]

@srowen @cloud-fan @wbo4958 @HyukjinKwon @zhengruifeng

[cloud-fan]

thanks, merging to master!

[zhengruifeng]

late LGTM