Race in linecache under free-threading

[vfdev-5]

Bug report

Bug description:

There is a time of check to time of use race under free-threading in linecache.py implementation:

cpython/Lib/linecache.py

Lines 128 to 129 in ed039b8

	if filename in cache:
	if len(cache[filename]) != 1:

Here are two reproducers: one with linecache and second with traceback (a similar usage was originally seen in JAX CI)

1. Reproducer with linecache

import linecache
import concurrent.futures
import threading


if __name__ == "__main__":
    num_workers = 20
    num_runs = 100

    values = range(10, 15)
    for i in values:
        with open(f"test_{i}.py", "w") as h:
            h.write("import time\n")
            h.write("import system\n")

    barrier = threading.Barrier(num_workers)

    def closure():
        barrier.wait()
        for _ in range(num_runs):
            for name in values:
                linecache.getline(f"test_{name}.py", 1)

    with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
        futures = []
        for i in range(num_workers):
            futures.append(executor.submit(closure))
        assert len(list(f.result() for f in futures)) == num_workers

This gives:

Traceback (most recent call last):
  File "/project/playground/cpython_checks/linecache_race/repro.py", line 28, in <module>
    assert len(list(f.result() for f in futures)) == num_workers
               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/playground/cpython_checks/linecache_race/repro.py", line 28, in <genexpr>
    assert len(list(f.result() for f in futures)) == num_workers
                    ~~~~~~~~^^
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/_base.py", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/thread.py", line 73, in run
    return fn(*args, **kwargs)
  File "/project/playground/cpython_checks/linecache_race/repro.py", line 22, in closure
    linecache.getline(f"test_{name}.py", 1)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/cpython-tsan/lib/python3.14t/linecache.py", line 26, in getline
    lines = getlines(filename, module_globals)
  File "/tmp/cpython-tsan/lib/python3.14t/linecache.py", line 39, in getlines
    return cache[filename][2]
           ~~~~~^^^^^^^^^^
KeyError: 'test_11.py'

2. Reproducer with traceback:

import traceback
import concurrent.futures
import threading


if __name__ == "__main__":
    num_workers = 20
    num_runs = 100

    barrier = threading.Barrier(num_workers)

    def closure():
        # import test_10

        barrier.wait()
        for _ in range(num_runs):
            try:
                raise RuntimeError("STOP")
            except RuntimeError as e:
                tb = traceback.extract_stack(e.__traceback__.tb_frame)

    with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
        futures = []
        for i in range(num_workers):
            futures.append(executor.submit(closure))
        assert len(list(f.result() for f in futures)) == num_workers

Gives the output:

Traceback (most recent call last):
  File "/project/playground/cpython_checks/linecache_race/repro_traceback.py", line 36, in closure
    raise RuntimeError("STOP")
RuntimeError: STOP

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/playground/cpython_checks/linecache_race/repro_traceback.py", line 44, in <module>
    assert len(list(f.result() for f in futures)) == num_workers
               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/playground/cpython_checks/linecache_race/repro_traceback.py", line 44, in <genexpr>
    assert len(list(f.result() for f in futures)) == num_workers
                    ~~~~~~~~^^
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/_base.py", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/thread.py", line 73, in run
    return fn(*args, **kwargs)
  File "/project/playground/cpython_checks/linecache_race/repro_traceback.py", line 38, in closure
    tb = traceback.extract_stack(e.__traceback__.tb_frame)
  File "/tmp/cpython-tsan/lib/python3.14t/traceback.py", line 264, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/tmp/cpython-tsan/lib/python3.14t/traceback.py", line 457, in extract
    return klass._extract_from_extended_frame_gen(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        extended_frame_gen(), limit=limit, lookup_lines=lookup_lines,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        capture_locals=capture_locals)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/cpython-tsan/lib/python3.14t/traceback.py", line 508, in _extract_from_extended_frame_gen
    f.line
  File "/tmp/cpython-tsan/lib/python3.14t/traceback.py", line 377, in line
    self._set_lines()
    ~~~~~~~~~~~~~~~^^
  File "/tmp/cpython-tsan/lib/python3.14t/traceback.py", line 355, in _set_lines
    line = linecache.getline(self.filename, lineno).rstrip()
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/cpython-tsan/lib/python3.14t/linecache.py", line 26, in getline
    lines = getlines(filename, module_globals)
  File "/tmp/cpython-tsan/lib/python3.14t/linecache.py", line 42, in getlines
    return updatecache(filename, module_globals)
  File "/tmp/cpython-tsan/lib/python3.14t/linecache.py", line 129, in updatecache
    if len(cache[filename]) != 1:
           ~~~~~^^^^^^^^^^
KeyError: '/tmp/cpython-tsan/lib/python3.14t/concurrent/futures/thread.py'

Context: failure in JAX CI, https://github.com/jax-ml/jax/actions/runs/14770168501/job/41468833801#step:18:4022
@hawkinsp

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-133253: making linecache thread-safe #133305

[rhettinger]

@colesbury Is non-threadsafe pure python code considered a bug? My understanding was that no pure python code was considered thread-safe unless explicitly documented as such (the pure python versions of lru_cache and Queue for example).

Possibly, the linecache module may be a special case and need stricter requirements because it is foundational code for a number of other low level modules.

[colesbury]

I probably wouldn't consider this a freethreading bug, because the Python code isn't thread-safe even with the GIL, although the errors probably happen a lot more frequently in the free threading build.

I think it's basically a design question whether we want any specific Python module or class should be thread-safe. I think it's probably worth making linecache thread-safe, but not the Python versions of lru_cache or Queue:

• lru_cache and Queue have thread-safe C implementations that are typically used, so making them thread-safe is a bunch of work without clear benefit
• As you wrote, linecache is foundational and used by other modules
• I think it would be harder for users to work around the thread-unsafeness of linecache than lru_cache or Queue. There's only one global linecache module and you can't control accesses to it from code in other modules/packages.

So I'm not really sure if this is a bug or feature request, but that question seems less important to me than:

1. Would we like to make linecache thread-safe? (my vote is yes)
2. Do we want to prioritize the work? (I'm not going to work on a PR for 3.14, but I'm happy to help review someone else's PR.)

[vfdev-5]

Do we want to prioritize the work? (I'm not going to work on a PR for 3.14, but I'm happy to help review someone else's PR.)

I would be happy to send a fix if we can agree on a plan what to do. I think we can make it thread-safe using python threading locks.

[hawkinsp]

I suspect you can avoid Python-level locking on at least the reads if we are careful not to perform repeated lookups. e.g. consider:

cpython/Lib/linecache.py

Line 36 in ed039b8

if filename in cache:

    if filename in cache:
        entry = cache[filename]
        if len(entry) != 1:
            return cache[filename][2]

can be

   entry = cache.get(filename, None)
   if entry and len(entry) != 1:
     return entry[2]

which also saves you two dict lookups. The same idiom appears a few times.

[rhettinger]

I think it's probably worth making linecache thread-safe, but not the Python versions of lru_cache or Queue

@colesbury The opposite is currently true. lru_cache and Queue are implemented with locks are already thread-safe. It is linecache that has race conditions.

[colesbury]

@rhettinger - ah, I misunderstood your original comment. I still think it's worth making linecache thread safe.