Skip to content

[SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON #21659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 8 commits into from
Closed

[SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON #21659

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a1346e2
Add a control to force Python version via enviornment variable
HyukjinKwon Jun 28, 2018
dd303b6
Python change to trigger the stlye check
HyukjinKwon Jun 29, 2018
69f31a9
Revert "Python change to trigger the stlye check"
HyukjinKwon Jun 29, 2018
135d613
Address comments and correct its priority
HyukjinKwon Jul 2, 2018
71ff040
Fix some comments
HyukjinKwon Jul 2, 2018
950ead0
Run pydocstyle
HyukjinKwon Jul 9, 2018
d500e0d
Revert "Run pydocstyle"
HyukjinKwon Jul 9, 2018
2e1e7da
Remove changes in README.md which makes backporting harder
HyukjinKwon Jul 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions python/docs/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,44 @@
# Makefile for Sphinx documentation
#

ifndef SPHINXBUILD
ifndef SPHINXPYTHON
SPHINXBUILD = sphinx-build
endif
endif

ifdef SPHINXBUILD
Copy link
Contributor

@holdenk holdenk Jul 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to check that Sphinx build is running with Py3 as done bellow?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Jul 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find an easy way.

# User-friendly check for sphinx-build.
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
else
# Note that there is an issue with Python version and Sphinx in PySpark documentation generation.
# Please remove this check below when this issue is fixed. See SPARK-24530 for more details.
PYTHON_VERSION_CHECK = $(shell $(SPHINXPYTHON) -c 'import sys; print(sys.version_info < (3, 0, 0))')
Copy link
Contributor

@jerryshao jerryshao Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we fix the SPHINXPYTHON to python3 in release script release-build.sh?

Copy link
Contributor

@jerryshao jerryshao Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or add some options/outputs in release script to let others know how to workaround this issue.

Copy link
Member Author

@HyukjinKwon HyukjinKwon Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forcing SPHINXPYTHON to python3 by default will probably break the distribution builder in Jenkins if they are tried ... Seems there's an issue to force Sphinx to use Python 3 in Jenkins environment. This was the problem I struggled to tweak :(.

Am trying to update the release process - apache/spark-website#122. Would this be enough to address your concern?

ifeq ($(PYTHON_VERSION_CHECK), True)
$(error Note that Python 3 is required to generate PySpark documentation correctly for now. Current Python executable was less than Python 3. See SPARK-24530. To force Sphinx to use a specific Python executable, please set SPHINXPYTHON to point to the Python 3 executable.)
endif
# Check if Sphinx is installed.
ifeq ($(shell $(SPHINXPYTHON) -c 'import sphinx' >/dev/null 2>&1; echo $$?), 1)
$(error Python executable '$(SPHINXPYTHON)' did not have Sphinx installed. Make sure you have Sphinx installed, then set the SPHINXPYTHON environment variable to point to the Python executable having Sphinx installed. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Use 'SPHINXPYTHON -msphinx' instead of 'sphinx-build'. See https://github.com/sphinx-doc/sphinx/pull/3523 for more details.
SPHINXBUILD = $(SPHINXPYTHON) -msphinx
endif

# You can set these variables from the command line.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
PAPER ?=
BUILDDIR ?= _build
# You can set SPHINXBUILD to specify Sphinx build executable or SPHINXPYTHON to specify the Python executable used in Sphinx.
# They follow:
# 1. if SPHINXPYTHON is set, use Python. If SPHINXBUILD is set, use sphinx-build.
# 2. If both are set, SPHINXBUILD has a higher priority over SPHINXPYTHON
# 3. By default, SPHINXBUILD is used as 'sphinx-build'.

export PYTHONPATH=$(realpath ..):$(realpath ../lib/py4j-0.10.7-src.zip)

# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif

# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
Expand Down