Use HTTPS instead of HTTP

[svetlyak40wt]

Hi!

I've noticed that archives are downloaded via http:

CL-USER> (ql:quickload :thread-pool)
To load "thread-pool":
  Load 2 ASDF systems:
    arnesi bordeaux-threads
  Install 1 Quicklisp release:
    thread-pool
Downloading http://beta.quicklisp.org/archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz
##########################################################################

But I found that they are also available if I change schema to https:

[art@art-osx5:~]% curl -v -s https://beta.quicklisp.org/archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz > /dev/null
*   Trying 52.85.242.94...
* Connected to beta.quicklisp.org (52.85.242.94) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.quicklisp.org
* Server certificate: Gandi Standard SSL CA 2
* Server certificate: USERTrust RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET /archive/thread-pool/2012-01-07/thread-pool-20120107-git.tgz HTTP/1.1
> Host: beta.quicklisp.org
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 3061
< Connection: keep-alive
< Date: Sun, 03 Jun 2018 19:14:56 GMT
< Last-Modified: Sat, 07 Jan 2012 21:52:07 GMT
< ETag: "9dfcb3dd5692d474d90f7916722d5bf8"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 450
< X-Cache: Hit from cloudfront
< Via: 1.1 f9a0ddc3860252ab6c4d02ab024b4891.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: _WSj_KutsQ1kyoFACC3wQs3zq8jbyo5QjQXIcrCb1DJi4mCZKz2CVw==
<
{ [3061 bytes data]
* Connection #0 to host beta.quicklisp.org left intact

It would be great (and more secure) to switch to the HTTPS. What do you think, @xach?

[svetlyak40wt]

Here is some system info:

CL-USER> (cl-info:make-cl-info)
OS:   Darwin 15.6.0
Lisp: SBCL 1.4.3
ASDF: 3.3.1.1
QL:  org.borodust.bodge 20180214223017
     quicklisp 2017-10-23
CL-USER> (ql:update-client)
Downloading http://beta.quicklisp.org/client/quicklisp.sexp
##########################################################################
The most up-to-date client, version 2017-03-06, is already installed.
T
CL-USER> (ql:client-version)
"2017-03-06"

[quicklisp]

It would be good to do, but there's no straightforward path to do it. Implementations do not all provide HTTPS support, it's not straightforward to make it from scratch or use HTTPS libraries on all supported platforms.

I hope to soon roll out a different form of authentication of downloads via signatures.

[eugeneia]

Alternatively, for the time being I figure it is possible to point quicklisp at a http proxy that will convert requests to https, I think?

[svetlyak40wt]

Is there any news about a new form of authentication for quicklisp?

[Hexstream]

SHA256 verification of downloads or similar might be easier to implement than HTTPS?...

(Of course, both would be better eventually.)

[mohe2015]

SHA256 verification of downloads or similar might be easier to implement than HTTPS?...

(Of course, both would be better eventually.)

I don't know if I'm just stupid but I think SHA256 verification doesn't help without HTTPS. I think you need some kind of signature (like pgp) as @quicklisp already suggested

[Hexstream]

You communicate the SHA256 hashes in advance in a secure way authenticated by a signature (as part of the dist metadata download), and then later you download over HTTP (or HTTPS) and verify that the hashes of the files are the expected ones...

[mohe2015]

@Hexstream So yeah you meant a combination. I thought you meant to only use SHA256 sums.

[svetlyak40wt]

By the way, guys, have you seen an alternative package manager https://gitlab.common-lisp.net/clpm/clpm which is able to use a quicklisp distribution and to force HTTPS?

I've found it recently but didn't have time to try.

[ghard]

I guess integrity is more important than privacy in our case, so the common method would use something like a GPG key to sign the list of package checksums upon distro build. Then, once downloaded and verified, that list can be used to verify individual packages.

That would mean another external dependency and calling inferiors through OS. GPG binaries are readily available on most platforms though.

Otherwise one could depend on ironclad for PK cryptography and signing/verification of the package cksum list. One of GPG's "strengths" is that the distribution signing key could be acquired/verified trough already existing key server infrastructure.

[xach]

I have Quicklisp client code for sha256 checking and GPG signature checking in portable CL. See the gpg branch for more info.

Right now the hold up is key management and a bit of infrastructure updates.

[Hexstream]

I note in passing that Keybase is a pretty great modern alternative to the fairly antiquated GPG infrastructure.

[thephoeron]

One straightforward approach to handling HTTPS is graceful degradation. You already have HTTP connections with GPG signature checking as the fall-through. Adding platform-specific support is unpleasant, but it's either here or in a dedicated portable SSL library that will have to be bundled into Quicklisp.

From there, make it highly visible when using HTTPS. When not, keep the same interface we're used to.

Is that sufficient or am I overlooking anything?

[Risto-Stevcev]

Is this still in progress?
I created cl-micropm for some of Quicklisp's limitations (CLPM development seems to have stopped), but it would be nice to have support for HTTPS and Github/Gitlab sources. Or if there's a way to move out Quicklisp's dependency resolution algorithm so that we can build stuff on top of it -- I tried a couple of times to track it down and pull it out in the code but with no success.