`socketserver.Unix*Server` may attempt to set `SO_REUSEPORT` on Unix sockets, causing `EOPNOTSUPP`, e.g. in `test_asyncio`

[mgorny]

Bug report

Bug description:

When socketserver.Unix*Server.server_bind() is called with allow_reuse_port attribute set to True, it attempts to call self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) on a Unix socket, and it fails with: OSError: [Errno 95] Operation not supported. In particular, this happens in test_asyncio tests.

This seems to be a recent change on my system (probably in Linux kernel), as the relevant tests used to pass when I tested CPython 3.14.0a3, and now the same version failed. Now I'm at 6.12.9, back then it was 6.12.5 (but I don't think there's a point in actually verifying that).

Output from `test_asyncio.test_sock_lowlevel`

$ ./python -E  -m test -v test_asyncio.test_sock_lowlevel
== CPython 3.14.0a4+ (heads/main:3193cb5ef87, Jan 16 2025, 13:54:51) [GCC 14.2.1 20241221]
== Linux-6.12.9-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.40 little-endian
== Python build: release
== cwd: /home/mgorny/git/cpython/build/test_python_worker_532605æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3831794787
0:00:00 load avg: 0.54 Run 1 test sequentially in a single process
0:00:00 load avg: 0.54 [1/1] test_asyncio.test_sock_lowlevel
test_cancel_sock_accept (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_cancel_sock_accept) ... ok
test_create_connection_sock (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_create_connection_sock) ... ok
test_huge_content (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_huge_content) ... ok
test_huge_content_recvinto (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_huge_content_recvinto) ... ok
test_recvfrom (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_recvfrom) ... ok
test_recvfrom_into (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_recvfrom_into) ... ok
test_sendto_blocking (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sendto_blocking) ... ok
test_sock_accept (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sock_accept) ... ok
test_sock_client_connect_racing (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sock_client_connect_racing) ... ok
test_sock_client_fail (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sock_client_fail) ... ok
test_sock_client_ops (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sock_client_ops) ... ok
test_sock_client_racing (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_sock_client_racing) ... ok
test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_unix_sock_client_ops) ... ERROR
test_cancel_sock_accept (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_cancel_sock_accept) ... ok
test_create_connection_sock (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_create_connection_sock) ... ok
test_huge_content (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_huge_content) ... ok
test_huge_content_recvinto (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_huge_content_recvinto) ... ok
test_recvfrom (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_recvfrom) ... ok
test_recvfrom_into (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_recvfrom_into) ... ok
test_sendto_blocking (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sendto_blocking) ... ok
test_sock_accept (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sock_accept) ... ok
test_sock_client_connect_racing (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sock_client_connect_racing) ... ok
test_sock_client_fail (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sock_client_fail) ... ok
test_sock_client_ops (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sock_client_ops) ... ok
test_sock_client_racing (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_sock_client_racing) ... ok
test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_unix_sock_client_ops) ... ERROR
test_cancel_sock_accept (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_cancel_sock_accept) ... ok
test_create_connection_sock (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_create_connection_sock) ... ok
test_huge_content (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_huge_content) ... ok
test_huge_content_recvinto (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_huge_content_recvinto) ... ok
test_recvfrom (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_recvfrom) ... ok
test_recvfrom_into (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_recvfrom_into) ... ok
test_sendto_blocking (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sendto_blocking) ... ok
test_sock_accept (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sock_accept) ... ok
test_sock_client_connect_racing (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sock_client_connect_racing) ... ok
test_sock_client_fail (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sock_client_fail) ... ok
test_sock_client_ops (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sock_client_ops) ... ok
test_sock_client_racing (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_sock_client_racing) ... ok
test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_unix_sock_client_ops) ... ERROR

======================================================================
ERROR: test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.EPollEventLoopTests.test_unix_sock_client_ops)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/test_sock_lowlevel.py", line 462, in test_unix_sock_client_ops
    with test_utils.run_test_unix_server() as httpd:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 261, in run_test_unix_server
    yield from _run_test_server(address=path, use_ssl=use_ssl,
                                server_cls=SilentUnixWSGIServer,
                                server_ssl_cls=UnixSSLWSGIServer)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 188, in _run_test_server
    httpd = server_class(address, SilentWSGIRequestHandler)
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 457, in __init__
    self.server_bind()
    ~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 217, in server_bind
    UnixHTTPServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 207, in server_bind
    socketserver.UnixStreamServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 472, in server_bind
    self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 95] Operation not supported

======================================================================
ERROR: test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.PollEventLoopTests.test_unix_sock_client_ops)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/test_sock_lowlevel.py", line 462, in test_unix_sock_client_ops
    with test_utils.run_test_unix_server() as httpd:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 261, in run_test_unix_server
    yield from _run_test_server(address=path, use_ssl=use_ssl,
                                server_cls=SilentUnixWSGIServer,
                                server_ssl_cls=UnixSSLWSGIServer)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 188, in _run_test_server
    httpd = server_class(address, SilentWSGIRequestHandler)
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 457, in __init__
    self.server_bind()
    ~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 217, in server_bind
    UnixHTTPServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 207, in server_bind
    socketserver.UnixStreamServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 472, in server_bind
    self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 95] Operation not supported

======================================================================
ERROR: test_unix_sock_client_ops (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests.test_unix_sock_client_ops)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/test_sock_lowlevel.py", line 462, in test_unix_sock_client_ops
    with test_utils.run_test_unix_server() as httpd:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 261, in run_test_unix_server
    yield from _run_test_server(address=path, use_ssl=use_ssl,
                                server_cls=SilentUnixWSGIServer,
                                server_ssl_cls=UnixSSLWSGIServer)
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 188, in _run_test_server
    httpd = server_class(address, SilentWSGIRequestHandler)
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 457, in __init__
    self.server_bind()
    ~~~~~~~~~~~~~~~~^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 217, in server_bind
    UnixHTTPServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/test/test_asyncio/utils.py", line 207, in server_bind
    socketserver.UnixStreamServer.server_bind(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/mgorny/git/cpython/Lib/socketserver.py", line 472, in server_bind
    self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 95] Operation not supported

----------------------------------------------------------------------
Ran 39 tests in 1.296s

FAILED (errors=3)
test test_asyncio.test_sock_lowlevel failed
test_asyncio.test_sock_lowlevel failed (3 errors)

== Tests result: FAILURE ==

1 test failed:
    test_asyncio.test_sock_lowlevel

Total duration: 1.4 sec
Total tests: run=39
Total test files: run=1/1 failed=1
Result: FAILURE

Bisect points to 192d17c:

commit 192d17c3fd9945104bc0303cf248bb0d074d260e
Author:     Idan Kapustian <71190257+idankap@users.noreply.github.com>
AuthorDate: 2024-06-16 14:15:03 +0200
Commit:     GitHub <noreply@github.com>
CommitDate: 2024-06-16 14:15:03 +0200

    gh-120485: Add an override of `allow_reuse_port` on classes subclassing `socketserver.TCPServer` (GH-120488)
    
    Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-128916: Do not set SO_REUSEPORT on non-AF_INET* sockets #128933
• [3.13] gh-128916: Do not set SO_REUSEPORT on non-AF_INET* sockets (GH-128933) #128969
• [3.12] gh-128916: Do not set SO_REUSEPORT on non-AF_INET* sockets (GH-128933) #128970

[mgorny]

Indeed, here's the kernel commit: torvalds/linux@5b0af62

[mgorny]

CC @idankap @vsajip

Not sure if we should just ignore allow_reuse_port on Unix sockets, or assume it's invalid to set it to True, and modify the tests not to do that. Also, unclear on allow_reuse_address — it doesn't fail (right now), but no clue if it actually does anything.

[vsajip]

Yes, I suppose neither allow_reuse_XXX really makes sense for domain sockets. I guess if the change is done in socketserver.Unix*Server to ignore those settings when a domain socket is in use, nothing else needs to change?

[mgorny]

Yes, the following seems to fix the issue:

diff --git a/Lib/socketserver.py b/Lib/socketserver.py
index cd028ef1c63..d1d292c623f 100644
--- a/Lib/socketserver.py
+++ b/Lib/socketserver.py
@@ -722,9 +722,15 @@ class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass
     class UnixStreamServer(TCPServer):
         address_family = socket.AF_UNIX
 
+        allow_reuse_address = False
+        allow_reuse_port = False
+
     class UnixDatagramServer(UDPServer):
         address_family = socket.AF_UNIX
 
+        allow_reuse_address = False
+        allow_reuse_port = False
+
     class ThreadingUnixStreamServer(ThreadingMixIn, UnixStreamServer): pass
 
     class ThreadingUnixDatagramServer(ThreadingMixIn, UnixDatagramServer): pass

Should I make it into a pull request, or do you want to handle it from here? I suppose a comment noting that this overrides the value from http.server.HTTPServer may help too, given that the directly inherited TCPServer and UDPServer both default to False.

[vsajip]

I'm confused. I just looked at the main branch of the repository:

$ rg allow_reuse_port Lib | more
Lib/socketserver.py:    - allow_reuse_port
Lib/socketserver.py:    - allow_reuse_port
Lib/socketserver.py:    allow_reuse_port = False
Lib/socketserver.py:        if self.allow_reuse_port and hasattr(socket, "SO_REUSEPORT"):
Lib/socketserver.py:    allow_reuse_port = False
Lib/test/test_logging.py:    allow_reuse_port = True
Lib/xmlrpc/server.py:    allow_reuse_port = True
Lib/logging/config.py:        allow_reuse_port = True
Lib/http/server.py:    allow_reuse_port = True

Since TCPServer and UDPServer already set the allow_reuse_port class attribute to False, which value should be inherited by UnixStreamServer and UnixDatagramServer respectively, there should be no need to override those values as in your patch, right? I can't immediately see where the code is that sets that attribute to True which would explain the failure of test_asyncio.test_sock_lowlevel.

[mgorny]

That's because UnixHTTPServer inherits HTTPServer, and the True comes from there:

cpython/Lib/test/test_asyncio/utils.py

Lines 204 to 209 in ea6cc26

	class UnixHTTPServer(socketserver.UnixStreamServer, HTTPServer):
	def server_bind(self):
	socketserver.UnixStreamServer.server_bind(self)
	self.server_name = '127.0.0.1'
	self.server_port = 80

I suppose that alternatively we could override these attributes there directly (and on the other servers in that file). Do you want me to test and prepare a PR doing that instead?

[vsajip]

Another way of looking at it is this: the problem is caused by calling setsockopt() with SO_REUSEPORT on the wrong kind of socket. So perhaps the better fix is to check the socket type using getsockopt() and not make the setsockopt() call if it's a domain socket?

[mgorny]

I suppose that makes sense too. After all, the point of "allowing reuse" is well, not to block, and with unix sockets this is never a problem.

[mgorny]

FWICS SO_REUSEADDR is valid for at least a few different address families, so I'll focus on SO_REUSEPORT only. I'll have a PR shortly.

[mgorny]

Filed #128933

[vsajip]

Excellent, thank you very much.