problems with SSL host verification with proxy servers

[stevengj]

Multiple people in the last few hours have reported problems downloading artifacts on Windows systems using proxy servers: here and here. In both cases, a workaround was ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com".

Given how quickly this popped up after the 1.6 release, I'm guessing that this will be a common problem, so I'm filing this issue to keep track of progress on tracking down the source of the problem and fixing it.

[lrnv]

For the record, i want to say that setting http_proxy and https_proxy as environnement variables is also necessary. This can be done before launching julia in a shell, or directly in Julia (since no shell is attached to a julia session by default on windows); My startup.jl got updated with:

ENV["http_proxy"] = "http://<your_proxy>:<port>"
ENV["https_proy"] = "http://<your_proxy>:<port>"
ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com"

where the proxy values are the same as the ones in your .gitconfig.

[StefanKarpinski]

It's unclear to me that there's anything wrong here. This post is my best understanding of the issue. It seems like what's going on here is that people are on Windows machines behind MITM proxies that are not configured to allow Windows CRL check requests. The configuration that works is to configure the proxy via the http_proxy and https_proxy variables and to tell libcurl not to try to verify hosts with the JULIA_SSL_NO_VERIFY_HOSTS variable. We could turn certificate revocation checking off, but that's insecure. We could have an option to turn revocation checking off without disabling host verification altogether, but since this is only a problem when you're behind a MITM firewall that will be presenting fake certificates for verification anyway, that's not actually any more secure than just turning verification off altogether. There seem to be two solutions here:

1. Tell IT to configure their MITM proxy/firewall so that it handles Windows CRL requests;
2. Barring that, tell end users to export JULIA_SSL_NO_VERIFY_HOSTS="**".

Note that the https?_proxy setting is only necessary for non-transparent proxies. At many organizations, the proxy is transparent and you don't have to do that but the JULIA_SSL_NO_VERIFY_HOSTS may still be necessary.

[lrnv]

I agree I'm not sure that anything can be done here, apart from pointing users having these issues to right configuration parameters.

Behavior or snooping IT departments is, sadly, not open to pull requests ATM.

[StefanKarpinski]

This issue should be mitigated by #115, which switches Downloads to using best effort CRL checking instead of failing if the firewall prevents the CRL check from getting through. This is now included in Downloads 1.4.1 which is used on the Julia 1.6 release branch, so should be included in Julia 1.6.2 when it is released. If you could potentially try a Julia 1.6 nightly or build the release-1.6 branch from source and see if it solves this problem, that would be great. Otherwise, we'll see if this issue is resolved in the 1.6.2 release when it comes out.

[lrnv]

On Julia 1.6.0 and Downloads 1.4.1, it works correctly for me, but i still have to set ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com".
But it also did before the change to Downloads, so i'm not sure about the result.

[StefanKarpinski]

You shouldn't have to set that environment variable anymore as long the CA root certificate for the MITM proxy/firewall is installed in the system CA root store. How did you go about trying Julia 1.6.0 with Downloads 1.4.1? Since it's a stdlib, you can't just use pkg> to install a different version — you pretty much have to build the latest 1.6 Julia from source in order to make sure that Pkg is using the latest Downloads. If you're just testing Downloads directly, it's possible to load a dev copy of the package in the REPL and test that, but you need to edit the project file and change the UUID in order to be able to load it (effectively making it seem like a different package from the stdlib that's already built into the system image).

[StefanKarpinski]

Closing this, assuming it's been fixed.

[lewisl]

Not fixed on MacOS v 12.0.1 as of Nov. 9, 2021.

I am not using a Proxy Server.

[StefanKarpinski]

If you're not using a proxy, then it can't be the same issue. If you could open a new issue describing the different problem you're having then perhaps we could debug it. If you could also provide necessary debugging info like your Julia version, that would be helpful.

[lewisl]

You are correct. An unrelated problem with some similar messages.

I've removed conda installed python and re-installed Julia. My issue was conflicting artifacts for QT5. Removing Python from within ~/.julia eliminates the duplication. GR brings In the only artifact for QT.

New Mac environments result in some challenges for Python, too.