Manylinux2010 #182
Next
Next commit
Minimal usable merge of markrwilliams:manylinux2 into current pypa:ma…
…ster
- The dockerfiles build successfully.
- Compared to current master, there's a new intermediate image for
glibc. As the x86_64 build depends on it, this must have a predefined
name. In the PR the "naming prefix" is 'markrwilliams/manylinux2',
therefore it's assumed for all images.
- There's a problem with libcurl which has been (temporarily) resolved
with script which redirects yum. See docker/build_scripts/build.sh for
details.
The following script is proven to work:
pushd docker/glibc/
docker build -t markrwilliams/manylinux2:centos-6.9-no-vsyscall
popd
docker/build_scripts/prefetch.sh curl openssl
docker build -t markrwilliams/manylinux2:x86_64 \
-f docker/Dockerfile-x86_64 docker/
docker build -t markrwilliams/manylinux2:i686 \
-f docker/Dockerfile-i686 docker/- Loading branch information
commit 7f4765ea62aca5e80c9621a4b35d1c341c8d7e43
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| #!/usr/bin/python | ||
| """ | ||
| Emulate linux32, i.e. setarch(8). | ||
| """ | ||
| import os | ||
| import ctypes | ||
| import sys | ||
|
|
||
| # Retrieved from a 32-bit CentOS 6.9 installation's | ||
| # /usr/include/sys/personality.h header. | ||
| LINUX_32 = 0x0008 | ||
|
|
||
| # Grab libc from our process | ||
| process_namespace = ctypes.CDLL(None, use_errno=True) | ||
| # int personality(unsigned long persona); | ||
| __syscall_personality = process_namespace.personality | ||
| __syscall_personality.argtypes = [ctypes.c_ulong] | ||
| __syscall_personality.restype = ctypes.c_int | ||
|
|
||
|
|
||
| def personality(persona): | ||
| """ | ||
| Wrap behavior of personality(2). | ||
| """ | ||
| set_persona = __syscall_personality(persona) | ||
| if set_persona == -1: | ||
| errno = ctypes.get_errno() | ||
| OSError(errno, os.strerror(errno)) | ||
|
|
||
| return set_persona | ||
|
|
||
|
|
||
| personality(LINUX_32) | ||
|
|
||
| argv = sys.argv[1:] | ||
| if not argv: | ||
| os.execlp("/bin/sh", "-sh") | ||
| elif argv[0] in ('-h', '--h', '-help', '--help'): | ||
| print("""Usage: | ||
| linux32 [<program> [<argument>...]] | ||
|
|
||
| Change the reported architecture to 32 bits. | ||
| THIS IS JUST A STUB FOR BOOTSTRAPPING! | ||
| Please install utils-linux-ng for the real executable. | ||
| """) | ||
| elif argv[0].startswith('-'): | ||
| print("linux32: Unknown option") | ||
| print("linux32: Try `linux32 --help' for more information.") | ||
| else: | ||
| os.execvp(argv[0], argv) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| FROM centos:6.9 as centos-with-vsyscall | ||
|
|
||
| COPY ./build_scripts /build_scripts | ||
| RUN bash /build_scripts/rebuild-glibc-without-vsyscall.sh | ||
|
|
||
| FROM centos:6.9 | ||
| LABEL maintainer="The Manylinux project" | ||
|
|
||
| COPY --from=centos-with-vsyscall /rpms /rpms | ||
|
|
||
| RUN yum -y install /rpms/* && rm -rf /rpms |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| centos-6.9-no-vsyscall | ||
| ====================== | ||
|
|
||
| *Summary*: Because of | ||
| https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, | ||
| this a CentOS 6.9 Docker image that rebuilds ``glibc`` without | ||
| *vsyscall* is necessary to reliably run ``manylinux2`` on 64-bit | ||
|
|
||
| hosts. This requires building the image on a system with | ||
| ``vsyscall=emulate`` but allows the resulting container to run on | ||
| systems with ``vsyscall=none`` or ``vsyscall=emulate``. | ||
|
|
||
| *vsyscall* is an antiquated optimization for a small number of | ||
| frequently-used system calls. A vsyscall-enabled Linux kernel maps a | ||
| read-only page of data and system calls into a process' memory at a | ||
| fixed address. These system calls can then be invoked by | ||
| dereferencing a function pointers to fixed offsets in that page, | ||
| saving a relatively expensive context switch. [1]_ | ||
|
|
||
| Unfortunately, because the code and its location in memory are fixed | ||
| and well-known, the vsyscall mechanism has become a source of gadgets | ||
| for ROP attacks (specifically, Sigreturn-Oriented Programs). [2]_ | ||
| Linux 3.1 introduced vsyscall emulation that prevents attackers from | ||
| jumping into the middle of the system calls' code at the expense of | ||
| speed, as well as the ability to disable it entirely. [3]_ [4]_ The | ||
| vsyscall mechanism could not be eliminated at the time because | ||
| ``glibc`` versions earlier than 2.14 contained hard-coded references | ||
| to the fixed memory address, specifically in ``time(2)``. [5]_ These | ||
| segfault when attempting to issue a vsyscall-optimized system call | ||
| against a kernel that has disabled it. | ||
|
|
||
| Linux introduced a "virtual dynamic shared object" (vDSO) that | ||
| achieves the same high-speed, in-process system call mechanism via | ||
| shared objects sometime before the kernel's migration to git. While | ||
| old itself, vDSO 's presentation as a shared library allows it to | ||
| benefit from ASLR on modern systems, making it no more amenable to ROP | ||
| gadgets than any other shared library. ``glibc`` only switched over | ||
| completely to vDSO as of glibc 2.25, so until recently vsyscall | ||
| emulation has remained on for most kernels. [6]_ Furthermore, i686 | ||
| does not use vsyscall at all, so no version of ``glibc`` requires | ||
| patching on that architecture. | ||
|
|
||
| At the same time, vsyscall emulation still exposed values useful to | ||
| ROP attacks, so Linux 4.4 added a compilation option to disable | ||
| it. [7]_ [8]_ Distributions are beginning to ship kernels configured | ||
| without vsyscall, and running CentOS 5 (``glibc`` 2.5) or 6 (``glibc`` | ||
| 2.12) Docker containers on these distributions indeed causes segfaults | ||
| without ``vsyscall=emulate`` [9]_ [10]_. CentOS 6, however, is | ||
| supported until 2020. It is likely that more and more distributions | ||
| will ship with ``CONFIG_LEGACY_VSYSCALL_NONE``; if managed CI services | ||
| like Travis make this switch, developers will be unable to build | ||
| ``manylinux2`` wheels with our Docker image. | ||
|
|
||
| Fortunately, vsyscall is merely an optimization, and patches that | ||
| remove it can be backported to glibc 2.12 and the library recompiled. | ||
| The result is this Docker image. It can be run on kernels regardless | ||
| of their vsyscall configuration because executable and libraries on | ||
| CentOS are dynamically linked against glibc. Libraries built on this | ||
| image are unaffected because: | ||
|
|
||
| a) the kernel only maps vsyscall pages into processes; | ||
| b) only glibc used the vsyscall interface directly, and it's | ||
| included in manylinux2's whitelist policy. | ||
|
|
||
| Developers who build this vsyscall-less Docker image itself, however, | ||
| must do so on a system with ``vsyscall=emulate``. | ||
|
|
||
| References: | ||
| =========== | ||
|
|
||
| .. [1] https://lwn.net/Articles/446528/ | ||
| .. [2] http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf | ||
| .. [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5cec93c216db77c45f7ce970d46283bcb1933884 | ||
| .. [4] https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1 | ||
| .. [5] https://sourceware.org/git/?p=glibc.git;a=blob;f=ChangeLog;h=3a6abda7d07fdaa367c48a9274cc1c08498964dc;hb=356f8bc660a154a07b03da7c536831da5c8f74fe | ||
| .. [6] https://sourceware.org/git/?p=glibc.git;a=blob;f=ChangeLog;h=6037fef737f0338a84c6fb564b3b8dc1b1221087;hb=58557c229319a3b8d2eefdb62e7df95089eabe37 | ||
| .. [7] https://googleprojectzero.blogspot.fr/2015/08/three-bypasses-and-fix-for-one-of.html | ||
| .. [8] https://outflux.net/blog/archives/2016/09/27/security-things-in-linux-v4-4/ | ||
| .. [9] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620#20 | ||
| .. [10] https://github.com/CentOS/sig-cloud-instance-images/issues/62 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| [base-source] | ||
| name=CentOS-6.9 - Base SRPMS | ||
| baseurl=http://vault.centos.org/6.9/os/Source/ | ||
| gpgcheck=1 | ||
| gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 | ||
| priority=1 | ||
| enabled=1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| diff --git a/SPECS/glibc.spec b/SPECS/glibc.spec | ||
| index 9bd07c9..c389711 100644 | ||
| --- a/SPECS/glibc.spec | ||
| +++ b/SPECS/glibc.spec | ||
| @@ -1,6 +1,6 @@ | ||
| %define glibcsrcdir glibc-2.12-2-gc4ccff1 | ||
| %define glibcversion 2.12 | ||
| -%define glibcrelease 1.209%{?dist} | ||
| +%define glibcrelease 1.209.1%{?dist} | ||
| %define run_glibc_tests 1 | ||
| %define auxarches athlon sparcv9v sparc64v alphaev6 | ||
| %define xenarches i686 athlon | ||
| @@ -273,6 +273,7 @@ Patch240: glibc-rh1384281.patch | ||
| Patch241: glibc-rh1338673.patch | ||
| Patch242: glibc-rh1358015.patch | ||
| Patch243: glibc-rh1012343.patch | ||
| +Patch244: remove-vsyscall.patch | ||
|
|
||
| Buildroot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) | ||
| Obsoletes: glibc-profile < 2.4 | ||
| @@ -719,6 +720,7 @@ package or when debugging this package. | ||
| %patch241 -p1 | ||
| %patch242 -p1 | ||
| %patch243 -p1 | ||
| +%patch244 -E -p3 | ||
|
|
||
| # A lot of programs still misuse memcpy when they have to use | ||
| # memmove. The memcpy implementation below is not tolerant at | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| #!/bin/sh | ||
| # Prep script for x86_64 that recompiles glibc without vsyscalls. | ||
|
|
||
| # Stop at any error, show all commands | ||
| set -ex | ||
|
|
||
| # Locate the prep directory | ||
| MY_DIR=/$(dirname "${BASH_SOURCE[0]}") | ||
|
|
||
| # glibc versions | ||
| ORIGINAL_GLIBC_VERSION=2.12-1.209 | ||
| PATCHED_GLIBC_VERSION=2.12-1.209.1 | ||
|
|
||
| # Source RPM topdir | ||
| SRPM_TOPDIR=/root/rpmbuild | ||
|
|
||
| # Source RPM download directory | ||
| DOWNLOADED_SRPMS=/root/srpms | ||
|
|
||
| # Include the CentOS source RPM repository. | ||
| # https://bugs.centos.org/view.php?id=1646 | ||
| cp $MY_DIR/CentOS-source.repo /etc/yum.repos.d/CentOS-source.repo | ||
|
|
||
| # Extract and prepare the source | ||
| # https://blog.packagecloud.io/eng/2015/04/20/working-with-source-rpms/ | ||
| yum -y update | ||
| yum -y install yum-utils rpm-build | ||
| yum-builddep -y glibc | ||
| mkdir $DOWNLOADED_SRPMS | ||
| # The glibc RPM's contents are owned by mockbuild | ||
| adduser mockbuild | ||
| # yumdownloader assumes the current working directory | ||
| (cd $DOWNLOADED_SRPMS && yumdownloader --source glibc) | ||
| rpm -ivh $DOWNLOADED_SRPMS/glibc-$ORIGINAL_GLIBC_VERSION.el6.src.rpm | ||
| # Prepare the source by applying Red Hat and CentOS patches | ||
| rpmbuild -bp $SRPM_TOPDIR/SPECS/glibc.spec | ||
|
|
||
| # Copy the vsyscall removal patch into place | ||
| cp $MY_DIR/remove-vsyscall.patch $SRPM_TOPDIR/SOURCES | ||
| # Patch the RPM spec file so that it uses the vsyscall removal patch | ||
| (cd $SRPM_TOPDIR/SPECS && patch -p2 < $MY_DIR/glibc.spec.patch) | ||
|
|
||
| # Build the RPMS | ||
| rpmbuild -ba $SRPM_TOPDIR/SPECS/glibc.spec | ||
|
|
||
| mv $SRPM_TOPDIR/RPMS/* /rpms/ |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should be concerned about this, but note that this doesnot correspond to centos 6.10. I'm not sure what the canonical location for the extended package for enterpriselinux (EPEL) rpm is, myself.