gh-99730: HEAD should remain HEAD request on redirect

[haampie]

• Issue: HEAD requests should be HEAD requests upon redirect #99730

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[ghost]

All commit authors signed the Contributor License Agreement.

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[encukou]

This makes sense to me.

[haampie]

Is there a way to test whether this is breaking in practice?

For example, I've noticed that a Gitlab instance (forgot which) had download URLs for release assets, that returned a redirect to an AWS bucket with a URL that contains a signature. What's signed is the request as a whole, including the HTTP verb. It would only work with GET, HEAD requests would error with 403.

So, following the redirect required a GET request.

Not sure whose bug that is.

[encukou]

It seems that the error is with Gitlab/AWS, but easily worked around by not doing HEAD requests to a service that doesn't support them.

@orsenthil, as the urllib expert, do you have an opinion?

[orsenthil]

do you have an opinion?
This change sounds reasonable and correct to me. I will find out why it was GET prior to this change and check the RFCs before approving and merging.

[haampie]

Looks like it's complicated.

The RFCs suggest "user interaction" in certain cases which is obviously impossible.

And there's different behavior for different types of 3xx status codes.

Looks like they realized that redirect on POST request is somewhat ambiguous: either it means the POST request has to be sent elsewhere, or, the POST request succeeded, you just have to GET the (success) response elsewhere. The 303 message resolves that ambiguity. I have no clue if any modern webserver is following the RFC correctly, they might still reply with 302 on success (?) -- the POST behavior in urllib makes sense to me.

From RFC 2616 10.3.2: 301 Moved Permanently

Note: When automatically redirecting a POST request after
receiving a 301 status code, some existing HTTP/1.0 user agents
will erroneously change it into a GET request.

From RFC 2616 10.3.3: 302 Found

Note: RFC 1945 and RFC 2068 specify that the client is not allowed
to change the method on the redirected request. However, most
existing user agent implementations treat 302 as if it were a 303
response, performing a GET on the Location field-value regardless
of the original request method. The status codes 303 and 307 have
been added for servers that wish to make unambiguously clear which
kind of reaction is expected of the client.

And RFC 2616 10.3.4: 303 See Other

The response to the request can be found under a different URI and
SHOULD be retrieved using a GET method on that resource. This method
exists primarily to allow the output of a POST-activated script to
redirect the user agent to a selected resource.

And RFC 2626 10.3.8: 307 Temporary Redirect

If the 307 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.

[encukou]

IMO, while it's not clear whether the redirected request should be made, it's pretty clear that if it is made, HEAD should stay HEAD rather than turn into GET.
As I read it, the standard's wording on 303 See Other says what to do to retrieve the resource. But making a HEAD request means you don't want to retrieve the resource :)

The behaviour on 307 Temporary Redirect does not follow the standard, but, that's not what this PR is about.

[encukou]

@orsenthil,

This change sounds reasonable and correct to me. I will find out why it was GET prior to this change and check the RFCs before approving and merging.

Is there anything I can do to help with this?

[encukou]

I will find out why it was GET prior to this change and check the RFCs before approving and merging.

@orsenthil Do you think you can find the time before Python 3.13 beta1?

[encukou]

I plan to merge this around Wednesday if there are no objections.

[orsenthil]

LGTM

[orsenthil]

@encukou -

I apologize, I forgot to get back to this issue and discussion. I spent time reviewing the RFCs, although it surprises me that we have this behavior (of HEAD upgrading to GET) for a long time, I was not able to find a requirement to this behavior in the RFCs. Further, the clients like curl, which I often used as a reference, do no upgrade HEAD to GET. So this change is correct thing to do.

There is a possibility of some client relying on this archaic behavior, but they wont be adversely affected, and they could handle this a GET again.

Thanks for change, and pushing this through.

[encukou]

Thank you for checking!