SSL issues with OSX on Azure

[djhoese]

A contributor to my project recently enabled a test that attempts to download a resource using HTTPS. It seems that running these tests on Azure fails for the OSX environments we have with errors like:

 (<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)>)

Which leads to issues on github like scikit-learn/scikit-learn#10201 (comment). It seems the "proper" solution is to properly install the SSL certificates that come with the python installation: https://bugs.python.org/issue29480 which can apparently be fixed with:

sudo /Applications/Python\ 3.6/Install\ Certificates.command

I'm not really familiar with how cibuildwheel is using the base image from the build environment, but is this something I should be able to control in my Azure environments? Is this something that can and should be fixed in cibuildwheel? Thanks for any help.

I see there are issues already related to SSL stuff, but they all seem a little different than this or say that this should already be fixed. I must be missing something.

[Czaki]

A contributor to my project recently enabled a test that attempts to download a resource using HTTPS. It seems that running these tests on Azure fails for the OSX environments we have with errors like:

Could you provide information about cibuildwheel version, python version (running test)? Which macos version?

[djhoese]

Yes, sorry. Of course.

Here is the full azure config if necessary: https://github.com/vispy/vispy/blob/master/azure-pipelines.yml

The two OSX systems are both OSX 10.14, but one uses the default build target while the other does MACOSX_DEPLOYMENT_TARGET="10.14" before calling cibuildwheel.

Our currently build skips are:

  CIBW_SKIP: "cp27-* cp34-* cp35-* cp39-* pp27-* pp36-*"

So python 3.6, 3.7, and 3.8 should be built.

Edit: And this should be using the latest cibuildwheel release (1.6.1).

[Czaki]

As project is public. Could You provide link to failing build?

[djhoese]

Let me know if you can see these: https://dev.azure.com/vispy/vispy/_build/results?buildId=574&view=results

[YannickJadoul]

Ping @mayeut, who's been patching SSL in older Python versions for cibuildwheel before.

[Czaki]

@djhoese I see it.
Could You ignore cp36 and check if everything pass?

It looks like outdated certificate center.
Maybe wee need to add below call to install python on macos

sudo /Applications/Python\ 3.6/Install\ Certificates.command

Did we know any MacPython expert who could tell us for which version of python this should be called?

[djhoese]

I could, but I had our internal pytest test get skipped for python 3.6 and it then failed for python 3.7. My understanding based on the python issue mentioned above is that this is the intended way for Python 3+ to work now with regards to certificate installation on OSX.

[mayeut]

Here's what I get for a list of CPython versions:

find /Applications/Python* -name 'Inst*'
/Applications/Python 2.7/Install Certificates.command
/Applications/Python 3.6/Install Certificates.command
/Applications/Python 3.7/Install Certificates.command
/Applications/Python 3.8/Install Certificates.command
/Applications/Python 3.9/Install Certificates.command

Only Python 3.5 doesn't come with that script but probably should (since it reports location of cert file like the others):

Matt$ /Library/Frameworks/Python.framework/Versions/3.5/bin/python3
Python 3.5.4 (v3.5.4:3f56838976, Aug  7 2017, 12:56:33) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Python 3.7.5 (v3.7.5:5c02a39a0b, Oct 14 2019, 19:07:34) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.8/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.9/bin/python3
Python 3.9.0 (v3.9.0:9cf6752276, Oct  5 2020, 11:29:23) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.9/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 16:07:15) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/2.7/etc/openssl/cert.pem'
>>> ^D

The script does the same thing for all versions, install/upgrade certifi, overwrite the ssl.get_default_verify_paths().openssl_cafile by a symlink to certifi.where()

[YannickJadoul]

@mayeut, thanks! But how comes we're only bumping into this now? What changed? Is this purely an issue with the Azure image?

EDIT: Nevermind. Ofc, it's triggered by "a test that attempts to download a resource using HTTPS"...

[Czaki]

@YannickJadoul May you Ask PyPy team how it looks in their case?

[YannickJadoul]

What does PyPy have to do with this?

[djhoese]

This should only be a problem (I think) if someone is making a request to an HTTPS resource from urllib (or maybe any python based library). Maybe it was just "luck" that no one ever tried doing this while building wheels. 🤷

[YannickJadoul]

@djhoese, yes, I've not been following the discussion in too much detail, but that would make sense.

Another question, then: how comes this isn't an issue when normal macOS users install Python from Python.org to their machine?

[Czaki]

What does PyPy have to do with this?

What is their mechanis to deal with certificates? If they use system or like in CPython they vendor in it.