Add docs about using Shiv for packaging (similar to PEX)

[grantjenks]

What changes were proposed in this pull request?

The Shiv packaging tool works similarly to PEX and can be used to distribute Python with its dependencies in an executable. These changes mention Shiv and demonstrate it's use similar to the PEX project.

Why are the changes needed?

Shiv is a widely used packaging tool similar to PEX. These changes mention Shiv as an alternative tool.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

No code changes.

[HyukjinKwon]

Let's create another section instead of reusing PEX, and make sure testing on Yarn cluster/client, K8S cluster/client and standalone modes (all options here are manually tested by me).
I don't have a strong opinion on this (and I have less context on Shiv). I am fine with adding this if this library is expected to be actively maintained. Asking to user mailing list and collecting more feedback from the community might be another option before we go ahead.

cc @mridulm and @xkrogen who might have a better context on this. cc @fhoering for PEX.

[HyukjinKwon]

ok to test

[HyukjinKwon]

for the PR itself,

1. mind enabling Github Actions in your forked repository? see also https://github.com/apache/spark/pull/34083/checks?check_run_id=3693066509
2. please file a JIRA and format the Pr title (see also https://spark.apache.org/contributing.html)

[SparkQA]

Test build #143578 has finished for PR 34083 at commit c93ec6f.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48087/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48087/

[mridulm]

Agree with @HyukjinKwon - it would be good to start a discussion in spark mailing lists as well.

[mridulm]

+CC @zhouyejoe

[zhouyejoe]

Hi, @HyukjinKwon. Thanks for reviewing it. We have tested in YARN cluster mode. Will check with client mode, and also K8s cluster/client mode, standalone mode.
I am wondering what kind of testing that needs to be conducted. Does a simple job with some dependencies work? Or do we need some other kinds complex set up?

[HyukjinKwon]

Just a simple job is good enough. E.g., running a pandas udf example mentioned in the docs.

BTW, let's make a separate section instead of reusing PEX in the docs.

[grantjenks]

Thanks @HyukjinKwon for reviewing. I will update the change to discuss Shiv in a separate section.

Regarding the GitHub Actions setup. I think I have them enabled:

But for some reason they're not triggering.

[HyukjinKwon]

did you face something like #32400 (comment)? otherwise rebasing would retrigger the build properly.

[grantjenks]

Yes, I missed that. I've clicked Ok on the /actions page and rebased my change to trigger the build. Thank you.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48176/

[SparkQA]

Test build #143663 has finished for PR 34083 at commit ad978fe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48176/

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[mridulm]

Looks like this got dropped from our review radar @HyukjinKwon.
Do you think this is valid ? If yes, we can reopen and review ...

[zhouyejoe]

@mridulm I think we can reopen this until we have done enough testing around using Shiv here. Thoughts?

[mridulm]

Sounds good, we can revisit the PR once we have validated things.

[zero323]

Sounds good, we can revisit the PR once we have validated things.

Shall we add a JIRA ticket for this?