[SPARK-43305] Add support for java 17 from spark 3.5.0 #56

vakarisbk · 2023-10-13T13:39:02Z

What changes were proposed in this pull request?

Create Java17 base images alongside Java11 images starting from spark 3.5.0
Change ubuntu version to 22.04 for scala2.12-java17-*

Why are the changes needed?

Spark supports multiple Java versions, but the images are currently built only with Java 11.

Does this PR introduce any user-facing change?

New images would be available in the repositories.

How was this patch tested?

3.5.0/scala2.12-java11-python3-r-ubuntu/Dockerfile

add-dockerfiles.sh

.github/workflows/test.yml

tools/template.py

vakarisbk · 2023-10-17T06:52:22Z

It seems to me that the builds are failing due to insufficient storage on the runners.

[info]   java.lang.AssertionError: assertion failed: Failed to execute -- bash -c MINIKUBE_IN_STYLE=true minikube status  --
[info] minikube
[info] type: Control Plane
[info] host: InsufficientStorage
[info] kubelet: Running
[info] apiserver: Running
[info] kubeconfig: Configured
[info] docker-env: in-use

Maybe it's possible to try switching to larger runners?

Yikun · 2023-10-17T07:16:09Z

I never hit this before, but if it is storage limit, you could try to remove some tmp file to save space, such as:

https://github.com/apache/spark-docker/blob/master/.github/workflows/main.yml#L249

sudo install minikube-linux-amd64 /usr/local/bin/minikube
rm minikube-linux-amd64

It's about to save 80MB

(later upadte) I also noticed this change also apply on main repo: https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L1045

(if step 1 is ok, we don't need this step)

spark-docker/.github/workflows/main.yml

Line 201 in 028efd4

fetch-depth: 0

remove fetch-depth: 0, seems also save some space?

3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile

vakarisbk · 2023-10-17T09:47:21Z

I never hit this before, but if it is storage limit, you could try to remove some tmp file to save space, such as:

https://github.com/apache/spark-docker/blob/master/.github/workflows/main.yml#L249
sudo install minikube-linux-amd64 /usr/local/bin/minikube
rm minikube-linux-amd64
It's about to save 80MB

(later upadte) I also noticed this change also apply on main repo: https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L1045

(if step 1 is ok, we don't need this step)

spark-docker/.github/workflows/main.yml

Line 201 in 028efd4

fetch-depth: 0

remove fetch-depth: 0, seems also save some space?

added rm minikube-linux-amd64 and removed accidental changes in 3.3.0. Now only 3.5.0 will be built.

removing fetch-deph: 0 would help with space, but the default is fetch-deph: 1 which only fetches a single commit form the main/master branch. That would make the 3.3.0 build fail as it needs to cherry-pick commits from history.
I've tested the removal of fetch-deph:0 with 3.3.0 on my repo: https://github.com/vakarisbk/spark-docker/actions/runs/6545230049/job/17773271828

vakarisbk · 2023-10-17T14:12:10Z

made fetch-depth: 0 applicable only for 3.3.0 (saves about 400MB for other builds)
borrowed CI runner cleanup scripts from the main spark repo free_disk_space_container free_disk_space and added an action step to execute them.
These scripts made the CI runner filesystem go from this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   66G   18G  80% /
tmpfs           3.4G  172K  3.4G   1% /dev/shm
tmpfs           1.4G  1.2M  1.4G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb15      105M  6.1M   99M   6% /boot/efi
/dev/sda1        14G  4.1G  9.0G  31% /mnt
tmpfs           693M   12K  693M   1% /run/user/1001

to this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   29G   55G  35% /
tmpfs           3.4G  172K  3.4G   1% /dev/shm
tmpfs           1.4G  1.2M  1.4G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb15      105M  6.1M   99M   6% /boot/efi
/dev/sda1        14G  4.1G  9.0G  31% /mnt
tmpfs           693M   12K  693M   1% /run/user/1001

@Yikun could you trigger the builds one more time?

vakarisbk · 2023-10-18T11:39:16Z

All tests have passed except the 3.3.2 build.

The 3.3.2 build fails due to an issue with the GPG key on keys.openpgp.org (key on keyserver.ubuntu.com works fine)

- gpg --keyserver hkps://keys.openpgp.org --recv-key "C56349D886F2B01F8CAE794C653C2301FEA493EE" 

gpg: key 653C2301FEA493EE: no user ID
gpg: Total number processed: 1 

- gpg --batch --verify spark.tgz.asc spark.tgz

gpg: Signature made Pn Vas 10 22:40:58 2023 EET
gpg:                using RSA key C56349D886F2B01F8CAE794C653C2301FEA493EE
gpg:                issuer "[email protected]"
gpg: Can't check signature: No public key

But that's probably not relevant to this PR?

Apart from that, the PR is ready for review.

Yikun · 2023-10-21T23:11:58Z

@viirya Hi, Would you mind taking a look on 3.3.2 release key issue. It might needs your help to upload the public key to opengpg, see [1] as ref.

[1] #55 (comment)

viirya · 2023-10-21T23:24:55Z

@Yikun Just uploaded to openpgp.

Yikun

LGTM except some tiny nits, Please also make sure:

All dockerfiles and entrypoint.sh should be generated by add-dockerfiles.sh
It would be better if you can publish these images in your local repo to test (by appending a local change line in your local branch .github/workflows/publish.yml L50), It's just a test but shouldn't be changed in this PR.

Thanks for your efforts, I believe this PR will be merged soon.

versions.json

testing/testing.sh

vakarisbk · 2023-10-22T15:43:23Z

Please also make sure:

All dockerfiles and entrypoint.sh should be generated by add-dockerfiles.sh

All 3.5.0 dockerfiles and entrypoints were generated using the add-dockerfiles.sh
To validate, I ran this diff:

mv 3.5.0 3.5.0_copy; \
./add-dockerfiles.sh 3.5.0; \
diff -r 3.5.0 3.5.0_copy;

It would be better if you can publish these images in your local repo to test (by appending a local change line in your local branch .github/workflows/publish.yml L50), It's just a test but shouldn't be changed in this PR.

I've published the images in my forked repository.
Publish job logs can be found here.

Let me know if anything else is needed

Yikun

LGTM, only one minor nit for me.

I also use tools/manifest.py manifest to check the versions.json, looks good! Thanks for your efforts!

$ tools/manifest.py manifest
Maintainers: Apache Spark Developers <[email protected]> (@ApacheSpark)
GitRepo: https://github.com/apache/spark-docker.git

Tags: 3.5.0-scala2.12-java17-python3-ubuntu, 3.5.0-java17-python3, 3.5.0-java17, python3-java17
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java17-python3-ubuntu

Tags: 3.5.0-scala2.12-java17-r-ubuntu, 3.5.0-java17-r
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java17-r-ubuntu

Tags: 3.5.0-scala2.12-java17-ubuntu, 3.5.0-java17-scala
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java17-ubuntu

Tags: 3.5.0-scala2.12-java17-python3-r-ubuntu
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java17-python3-r-ubuntu

Tags: 3.5.0-scala2.12-java11-python3-ubuntu, 3.5.0-python3, 3.5.0, python3, latest
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java11-python3-ubuntu

Tags: 3.5.0-scala2.12-java11-r-ubuntu, 3.5.0-r, r
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java11-r-ubuntu

Tags: 3.5.0-scala2.12-java11-ubuntu, 3.5.0-scala, scala
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java11-ubuntu

Tags: 3.5.0-scala2.12-java11-python3-r-ubuntu
Architectures: amd64, arm64v8
GitCommit: 277235945e8b9774cb927dd0f4bffdac38908352
Directory: ./3.5.0/scala2.12-java11-python3-r-ubuntu

Yikun · 2023-10-23T23:58:54Z

add-dockerfiles.sh

+scala2.12-java17-python3-r-ubuntu
+scala2.12-java17-python3-ubuntu
+scala2.12-java17-r-ubuntu
+scala2.12-java17-ubuntu


Because we only add after 3.5 version, so we should skip 3.3 / 3.4 version. So seems we need some thing like below:

if ! echo $VERSION | grep -Eq "^3.3|^3.4"; then TAGS+=" scala2.12-java17-python3-r-ubuntu scala2.12-java17-python3-ubuntu scala2.12-java17-r-ubuntu scala2.12-java17-ubuntu " fi

Yikun · 2023-10-24T00:07:06Z

cc @HyukjinKwon @zhengruifeng Would you mind also taking a look?

vakarisbk · 2023-11-07T09:40:44Z

@Yikun maybe we can have this PR merged even without @HyukjinKwon and @zhengruifeng approval? This will not impact the existing images

RoeiA · 2023-11-08T13:00:37Z

Waiting for this one for a while now, thank you @vakarisbk for this PR, would love to see it merged!

Yikun · 2023-11-09T01:46:26Z

Thanks for your efforts @vakarisbk , I'm going to merge this PR later today or tomorrow.

Yikun · 2023-11-10T03:36:56Z

@vakarisbk Merged to master.

Thanks all!

Yikun · 2023-11-10T03:38:47Z

Test publish on: https://github.com/apache/spark-docker/actions/runs/6820460339

Yikun · 2023-11-10T10:27:02Z

Image published on GHCR @vakarisbk, would you mind doing a post validation:

https://github.com/apache/spark-docker/pkgs/container/spark-docker%2Fspark

Then you could feel free to open a PR on official image like to add java17 images:
Update spark version to 3.5.0 docker-library/official-images#15363

The content can be generated by tools/manifest.py manifest.

vakarisbk changed the title ~~[WIP] Add support for java 17 and explicit Python versions from 3.5.0~~ [WIP] Add support for java 17 and explicit Python versions from spark 3.5.0 onwards Oct 13, 2023

vakarisbk marked this pull request as ready for review October 13, 2023 14:17

vakarisbk marked this pull request as draft October 13, 2023 14:18

Yikun reviewed Oct 13, 2023

View reviewed changes

3.5.0/scala2.12-java11-python3-r-ubuntu/Dockerfile Outdated Show resolved Hide resolved

add-dockerfiles.sh Show resolved Hide resolved

add-dockerfiles.sh Outdated Show resolved Hide resolved

Yikun mentioned this pull request Oct 13, 2023

[SPARK-40513] Switch 3.4.0 default Java to Java17 #35

Closed

Yikun reviewed Oct 13, 2023

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

Yikun reviewed Oct 13, 2023

View reviewed changes

tools/template.py Show resolved Hide resolved

vakarisbk changed the title ~~[WIP] Add support for java 17 and explicit Python versions from spark 3.5.0 onwards~~ Add support for java 17 from spark 3.5.0 onwards Oct 15, 2023

vakarisbk changed the title ~~Add support for java 17 from spark 3.5.0 onwards~~ Add support for java 17 from spark 3.5.0 Oct 15, 2023

vakarisbk marked this pull request as ready for review October 15, 2023 11:05

Yikun reviewed Oct 17, 2023

View reviewed changes

3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile Outdated Show resolved Hide resolved

Add support for java17 from Spark 3.5.0

57edd2f

vakarisbk force-pushed the master branch from ac971ef to 57edd2f Compare October 18, 2023 11:32

Yikun reviewed Oct 22, 2023

View reviewed changes

versions.json Outdated Show resolved Hide resolved

versions.json Outdated Show resolved Hide resolved

testing/testing.sh Outdated Show resolved Hide resolved

vakarisbk added 2 commits October 22, 2023 13:52

fix typos

0c81c27

fix whitespace typo

2772359

Yikun approved these changes Oct 24, 2023

View reviewed changes

exclude java17 tags from versions 3.3.x and 3.4.x

b236385

Yikun approved these changes Nov 9, 2023

View reviewed changes

Yikun closed this in 6f68fe0 Nov 10, 2023

Yikun changed the title ~~Add support for java 17 from spark 3.5.0~~ [SPARK-43305] Add support for java 17 from spark 3.5.0 Nov 10, 2023

vakarisbk mentioned this pull request Nov 12, 2023

Add additional images to Apache Spark 3.5.0 docker-library/official-images#15697

Merged

[SPARK-43305] Add support for java 17 from spark 3.5.0 #56

[SPARK-43305] Add support for java 17 from spark 3.5.0 #56

Uh oh!

Conversation

vakarisbk commented Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vakarisbk commented Oct 17, 2023

Uh oh!

Yikun commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vakarisbk commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vakarisbk commented Oct 17, 2023

Uh oh!

vakarisbk commented Oct 18, 2023

Uh oh!

Yikun commented Oct 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Oct 21, 2023

Uh oh!

Yikun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vakarisbk commented Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yikun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

vakarisbk Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

Yikun commented Oct 24, 2023

Uh oh!

vakarisbk commented Nov 7, 2023

Uh oh!

RoeiA commented Nov 8, 2023

Uh oh!

Yikun commented Nov 9, 2023

Uh oh!

Yikun commented Nov 10, 2023

Uh oh!

Yikun commented Nov 10, 2023

Uh oh!

Yikun commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vakarisbk commented Oct 13, 2023 •

edited

Loading

Yikun commented Oct 17, 2023 •

edited

Loading

vakarisbk commented Oct 17, 2023 •

edited

Loading

Yikun commented Oct 21, 2023 •

edited

Loading

Yikun left a comment •

edited

Loading

vakarisbk commented Oct 22, 2023 •

edited

Loading

Yikun left a comment •

edited

Loading

Yikun commented Nov 10, 2023 •

edited

Loading