[SPARK-48664][FOLLOWUP] Update 4.0.0-preview1

[Yikun]

What changes were proposed in this pull request?

• Remove wget -nv -O KEYS https://downloads.apache.org/spark/KEYS and gpg --import KEYS;, it works but not meet the security concern from DOI.
• add add-dockerfiles.sh 4.0.0-preview1 support to address [SPARK-48664] Add Apache Spark 4.0.0-preview1 Dockerfiles #61 (comment)
• Fix versions.json: consider java17 as default version, so we can remove java17 tag.

Why are the changes needed?

update 4.0.0-preview1

Does this PR introduce any user-facing change?

new release

How was this patch tested?

./add-dockerfiles.sh 4.0.0-preview1 , no diff

[Yikun]

29.02 + gpg --batch --keyserver hkps://keys.openpgp.org --recv-key 4DC9676CEF9A83E98FCA02784D6620843CD87F5A
29.03 gpg: keybox '/tmp/tmp.NsK96iEJwQ/pubring.kbx' created
30.05 gpg: key 4D6620843CD87F5A: new key but contains no user ID - skipped
30.05 gpg: Total number processed: 1
30.05 gpg:           w/o user IDs: 1
30.05 + gpg --batch --verify spark.tgz.asc spark.tgz
32.10 gpg: Signature made Tue 28 May 2024 04:39:57 PM UTC
32.10 gpg:                using RSA key 4DC9676CEF9A83E98FCA02784D6620843CD87F5A
32.10 gpg:                issuer "[email protected]"
32.10 gpg: Can't check signature: No public key
------

cc @cloud-fan maybe need to add gpg key according #54 (comment)

[cloud-fan]

@Yikun Done, not sure how soon it will be available

[cloud-fan]

shall we tag the "all-in-one" image with this spark version only name?

[cloud-fan]

Or we think it's better to exclude R support by default?

[Yikun]

I'm neutral on this. As I mentioned only had some short discussion on it.

cc @zhengruifeng @HyukjinKwon @gengliangwang any idea?

[1] https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/edit?disco=AAAAf2TyFr0

[HyukjinKwon]

if this was the first version, I might have preferred to have all-in-one to have 4.0.0-preview1
but it's been like that already so I think it's fine

[cloud-fan]

+1

[Yikun]

Actually, 4.0.0-preview1 haven't been tagged in the published image (apache/spark) yet.

So, we can tag 4.0.0-preview1 on 4.0.0-preview1/scala2.13-java17-python3-r-ubuntu. that means the apache/spark:4.0.0-preview1/scala2.13-java17-python3-r-ubuntu will have a new alias tag 4.0.0-preview1.

[cloud-fan]

BTW, there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

[Yikun]

Done, not sure how soon it will be available

seems already work.

there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

Yes, we can! Now, only mentioned in spark-docker CONTRIBUTING.md

[Yikun]

there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

I added the doc, maybe you could give some review from release manager view. See 7bc0c58

[cloud-fan]

there is also a Publish (Java 11 only), shall we mention it?

[Yikun]

It seems you didn't notice the first line before "or"? java 17 already mentioned, :)

for 4.x, java 17; for 3.x, java 17 and java 11.

[cloud-fan]

shouldn't mention https://github.com/apache/spark-docker/blob/master/.github/workflows/publish-java11.yml ? Or we should remove this file if it's no longer in use.

[cloud-fan]

If other Spark versions (3.x) exclude R support by default, I think 4.0 should follow it.

[Yikun]

ok

[Yikun]

Merged. @HyukjinKwon @cloud-fan Thanks!