Running the test suite with --junit-xml and -R incorrectly reports refleaks

[pablogsal]

BPO	39253
Nosy	@vstinner, @zooba, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-07.23:54:53.841>
labels = ['tests', '3.9']
title = 'Running the test suite with --junit-xml and -R incorrectly reports refleaks'
updated_at = <Date 2020-01-08.17:31:56.794>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-01-08.17:31:56.794>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2020-01-07.23:54:53.841>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 39253
keywords = []
message_count = 9.0
messages = ['359561', '359562', '359563', '359606', '359608', '359610', '359611', '359612', '359615']
nosy_count = 3.0
nosy_names = ['vstinner', 'steve.dower', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39253'
versions = ['Python 3.9']

Linked PRs

• gh-83434: Disable XML in regrtest when -R option is used #117232
• [3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch #117250
• [3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch (GH-117250) #117251

[pablogsal]

./python -m test test_list -R : --junit-xml something
0:00:00 load avg: 0.52 Run tests sequentially
0:00:00 load avg: 0.52 [1/1] test_list
beginning 9 repetitions
123456789
.........
test_list leaked [798, 798, 798, 798] references, sum=3192
test_list leaked [345, 345, 345, 345] memory blocks, sum=1380
test_list failed

== Tests result: FAILURE ==

1 test failed:
test_list

Total duration: 3.4 sec
Tests result: FAILURE

[pablogsal]

This is going to be very noisy now that we are using --junit-xml in the buildbots...

[vstinner]

Oh. That explains these strange results on AMD64 RHEL7 Refleaks 3.8:

https://buildbot.python.org/all/#/builders/219/builds/41

argv: [b'make', b'buildbottest', b'TESTOPTS=-R 3:3 -u-cpu --junit-xml test-results.xml -j10 ${BUILDBOT_TESTOPTS}', b'TESTPYTHONOPTS=', b'TESTTIMEOUT=11700']

...
test_quopri leaked [168, 168, 168] references, sum=504
test_quopri leaked [75, 75, 75] memory blocks, sum=225
1:50:20 load avg: 1.06 Re-running test_future5 in verbose mode
test_future5 leaked [42, 42, 42] references, sum=126
test_future5 leaked [20, 20, 20] memory blocks, sum=60
...

The previous build is successful and it didn't use --junit-xml option:

https://buildbot.python.org/all/#/builders/219/builds/40

[zooba]

We can stop using --junit-xml if we want. GitHub Actions doesn't support displaying test results anyway, so once Azure Pipelines is switched off we'll just be reading the console all the time anyway (unless Buildbot has a way of rendering test results that I haven't noticed?)

[zooba]

My guess would be that the reference tracking needs some kind of awareness for logging results. The results are collected as regular Python objects and formatted later, as we need to calculate totals before writing the XML file.

I guess keeping those objects around looks like a leak?

[pablogsal]

We can stop using --junit-xml if we want. GitHub Actions doesn't support displaying test results anyway, so once Azure Pipelines is switched off we'll just be reading the console all the time anyway (unless Buildbot has a way of rendering test results that I haven't noticed?)

We are using --junit-xml on the buildbots to analyze failures and in the future we can use it to better display the summary, as is more reliable than parsing the longs.

I guess keeping those objects around looks like a leak?

Yeah, because it seems that we collect those objects in the innermost runner, while the refleak runner runs the whole "test suite" (the test file, like test_list.py) as one unit.

To be honest, the easiest fix here maybe is just deactivating the --junit and make a warning as in bpo-27103

[pablogsal]

deactivating the --junit

I mean deactivating the --junit *if* is used together with -R

[vstinner]

FYI test_io is skipped in Python 2.7 when using -R, because test_io is not reliable. Extract of Lib/test/test_io.py:

    if huntrleaks:
        # FIXME: bpo-31731: test_io hangs with --huntrleaks
        print("Warning: bpo-31731: test_io hangs with --huntrleaks: "
              "exclude the test")
        nottests.add('test_io')

I'm not shocked by disabling JUnit feature when using -R. But I suggest to raise an error and tune the buildbot configuration to not use it, rather than silently ignore the option.

[pablogsal]

But I suggest to raise an error and tune the buildbot configuration to not use it, rather than silently ignore the option.

I already configured the buildbots to not use --junit when running with =R