Skip to content

[SPARK-21773][BUILD][DOCS] Installs mkdocs if missing in the path in SQL documentation build #18984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
HyukjinKwon wants to merge 1 commit into from
Closed

[SPARK-21773][BUILD][DOCS] Installs mkdocs if missing in the path in SQL documentation build #18984

HyukjinKwon wants to merge 1 commit into from

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Aug 18, 2017

What changes were proposed in this pull request?

This PR proposes to install mkdocs by pip install if missing in the path. Mainly to fix Jenkins's documentation build failure in spark-master-docs. See https://amplab.cs.berkeley.edu/jenkins/job/spark-master-docs/3580/console.

It also adds mkdocs as requirements in docs/README.md.

How was this patch tested?

I manually ran jekyll build under docs directory after manually removing mkdocs via pip uninstall mkdocs.

Also, tested this in the same way but on CentOS Linux release 7.3.1611 (Core) where I built Spark few times but never built documentation before and mkdocs is not installed.

...
Moving back into docs dir.
Moving to SQL directory and building docs.
Missing mkdocs in your path, trying to install mkdocs for SQL documentation generation.
Collecting mkdocs
  Downloading mkdocs-0.16.3-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 574kB/s
Requirement already satisfied: PyYAML>=3.10 in /usr/lib64/python2.7/site-packages (from mkdocs)
Collecting livereload>=2.5.1 (from mkdocs)
  Downloading livereload-2.5.1-py2-none-any.whl
Collecting tornado>=4.1 (from mkdocs)
  Downloading tornado-4.5.1.tar.gz (483kB)
    100% |████████████████████████████████| 491kB 1.4MB/s
Collecting Markdown>=2.3.1 (from mkdocs)
  Downloading Markdown-2.6.9.tar.gz (271kB)
    100% |████████████████████████████████| 276kB 2.4MB/s
Collecting click>=3.3 (from mkdocs)
  Downloading click-6.7-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 2.8MB/s
Requirement already satisfied: Jinja2>=2.7.1 in /usr/lib/python2.7/site-packages (from mkdocs)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from livereload>=2.5.1->mkdocs)
Requirement already satisfied: backports.ssl_match_hostname in /usr/lib/python2.7/site-packages (from tornado>=4.1->mkdocs)
Collecting singledispatch (from tornado>=4.1->mkdocs)
  Downloading singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting certifi (from tornado>=4.1->mkdocs)
  Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)
    100% |████████████████████████████████| 358kB 2.1MB/s
Collecting backports_abc>=0.4 (from tornado>=4.1->mkdocs)
  Downloading backports_abc-0.5-py2.py3-none-any.whl
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python2.7/site-packages (from Jinja2>=2.7.1->mkdocs)
Building wheels for collected packages: tornado, Markdown
  Running setup.py bdist_wheel for tornado ... done
  Stored in directory: /root/.cache/pip/wheels/84/83/cd/6a04602633457269d161344755e6766d24307189b7a67ff4b7
  Running setup.py bdist_wheel for Markdown ... done
  Stored in directory: /root/.cache/pip/wheels/bf/46/10/c93e17ae86ae3b3a919c7b39dad3b5ccf09aeb066419e5c1e5
Successfully built tornado Markdown
Installing collected packages: singledispatch, certifi, backports-abc, tornado, livereload, Markdown, click, mkdocs
Successfully installed Markdown-2.6.9 backports-abc-0.5 certifi-2017.7.27.1 click-6.7 livereload-2.5.1 mkdocs-0.16.3 singledispatch-3.4.0.3 tornado-4.5.1
Generating markdown files for SQL documentation.
Generating HTML files for SQL documentation.
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: .../spark/sql/site
Moving back into docs dir.
Making directory api/sql
cp -r ../sql/site/. api/sql
            Source: .../spark/docs
       Destination: .../spark/docs/_site
      Generating...
                    done.
 Auto-regeneration: disabled. Use --watch to enable.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 18, 2017

cc @jkbradley and @srowen. Would you have some time to take a look for this one?

@SparkQA
Copy link

SparkQA commented Aug 18, 2017

Test build #80819 has finished for PR 18984 at commit 57abb1f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 20, 2017

Let me merge this one and see if the doc build passes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 20, 2017

Merged to master.

@asfgit asfgit closed this in 41e0eb7 Aug 20, 2017
@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 20, 2017 •
edited
Loading

It looks still failed. Now it looks during mkdocs installation - https://amplab.cs.berkeley.edu/jenkins/job/spark-master-docs/3588/consoleFull

Missing mkdocs in your path, trying to install mkdocs for SQL documentation generation.
Collecting mkdocs
  Downloading mkdocs-0.16.3-py2.py3-none-any.whl (1.2MB)
Collecting livereload>=2.5.1 (from mkdocs)
  Downloading livereload-2.5.1-py2-none-any.whl
Requirement already satisfied: Jinja2>=2.7.1 in /home/anaconda/lib/python2.7/site-packages (from mkdocs)
Collecting click>=3.3 (from mkdocs)
  Downloading click-6.7-py2.py3-none-any.whl (71kB)
Collecting Markdown>=2.3.1 (from mkdocs)
  Downloading Markdown-2.6.9.tar.gz (271kB)
Requirement already satisfied: PyYAML>=3.10 in /home/anaconda/lib/python2.7/site-packages (from mkdocs)
Collecting tornado>=4.1 (from mkdocs)
  Downloading tornado-4.5.1.tar.gz (483kB)
Requirement already satisfied: six in /home/anaconda/lib/python2.7/site-packages (from livereload>=2.5.1->mkdocs)
Collecting singledispatch (from tornado>=4.1->mkdocs)
  Downloading singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting certifi (from tornado>=4.1->mkdocs)
  Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)
Collecting backports_abc>=0.4 (from tornado>=4.1->mkdocs)
  Downloading backports_abc-0.5-py2.py3-none-any.whl
Building wheels for collected packages: Markdown, tornado
  Running setup.py bdist_wheel for Markdown: started
  Running setup.py bdist_wheel for Markdown: finished with status 'done'
  Stored in directory: /home/jenkins/.cache/pip/wheels/bf/46/10/c93e17ae86ae3b3a919c7b39dad3b5ccf09aeb066419e5c1e5
  Running setup.py bdist_wheel for tornado: started
  Running setup.py bdist_wheel for tornado: finished with status 'done'
  Stored in directory: /home/jenkins/.cache/pip/wheels/84/83/cd/6a04602633457269d161344755e6766d24307189b7a67ff4b7
Successfully built Markdown tornado
Installing collected packages: singledispatch, certifi, backports-abc, tornado, livereload, click, Markdown, mkdocs
Exception:
Traceback (most recent call last):
  File "/home/anaconda/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/anaconda/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/home/anaconda/lib/python2.7/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/home/anaconda/lib/python2.7/site-packages/pip/wheel.py", line 323, in clobber
    shutil.copyfile(srcfile, destfile)
  File "/home/anaconda/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/home/anaconda/lib/python2.7/site-packages/singledispatch_helpers.pyc
...

but I believe the fix itself is still okay and no need to revert.

Hi @shaneknapp, I guess, without a close look, we could simply install this by sudo pip install mkdocs. Maybe, would you please be able to check this and install mkdocs package for SQL documentation build?

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 21, 2017 via email

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 21, 2017

Here is the context I got:

#18702 broke the documentation build in https://amplab.cs.berkeley.edu/jenkins/job/spark-master-docs/.

...
Moving back into docs dir.
Moving to SQL directory and building docs.
Missing mkdocs in your path, skipping SQL documentation generation.
Moving back into docs dir.
Making directory api/sql
cp -r ../sql/site/. api/sql
jekyll 2.5.3 | Error:  unknown file type: ../sql/site/.
Deleting credential directory /home/jenkins/workspace/spark-master-docs/spark-utils/new-release-scripts/jenkins/jenkins-credentials-scUXuITy
Build step 'Execute shell' marked build as failure
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: FAILURE

It failed in this way roughly about 20ish days.

That PR adds SQL documentation build but with the dependency, mkdocs package. I completely forgot we actually build it in Jenkins.

To fix this, in this PR, I manually added some install command here to install mkdocs if missing in the path; however, it looks failed to install this with the error message:

...
Installing collected packages: singledispatch, certifi, backports-abc, tornado, livereload, click, Markdown, mkdocs
Exception:
Traceback (most recent call last):
  File "/home/anaconda/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/anaconda/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/home/anaconda/lib/python2.7/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/home/anaconda/lib/python2.7/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/home/anaconda/lib/python2.7/site-packages/pip/wheel.py", line 323, in clobber
    shutil.copyfile(srcfile, destfile)
  File "/home/anaconda/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/home/anaconda/lib/python2.7/site-packages/singledispatch_helpers.pyc
...

It has failed in this way roughly about 1 day.

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 21, 2017

okie dokie. installing mkdocs now.

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 21, 2017

done:

TASK [jenkins-worker : Install anaconda python packages via pip] ***************************************************************************
ok: [amp-jenkins-staging-worker-02.amp] => (item=discover)
ok: [amp-jenkins-staging-worker-01.amp] => (item=discover)
ok: [amp-jenkins-staging-worker-02.amp] => (item=fabric)
ok: [amp-jenkins-staging-worker-01.amp] => (item=fabric)
ok: [amp-jenkins-staging-worker-02.amp] => (item=findspark)
ok: [amp-jenkins-staging-worker-01.amp] => (item=findspark)
ok: [amp-jenkins-staging-worker-02.amp] => (item=flatbuffers)
ok: [amp-jenkins-staging-worker-01.amp] => (item=flatbuffers)
changed: [amp-jenkins-staging-worker-01.amp] => (item=mkdocs)
changed: [amp-jenkins-staging-worker-02.amp] => (item=mkdocs)
ok: [amp-jenkins-staging-worker-02.amp] => (item=subprocess32)
ok: [amp-jenkins-staging-worker-01.amp] => (item=subprocess32)

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 21, 2017

Nice! Thank you so much @shaneknapp!

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 21, 2017 via email

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 22, 2017 via email

@shaneknapp
Copy link
Contributor

shaneknapp commented Aug 22, 2017 via email

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Aug 22, 2017

Thanks for your effort @shaneknapp. I just checked the green.

@HyukjinKwon HyukjinKwon deleted the sql-doc-mkdocs branch January 2, 2018 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@HyukjinKwon @SparkQA @shaneknapp @srowen