cmd/go, testing: consider shrinking set of characters allowed/passed through in subtest names

[dmitshur]

This issue is meant to capture a small finding from #73086, and it's related to the cluster of proposals like #67562 (and its earlier version #45549), #27896, possibly more like #44970, #20209, #20210, #20115, and so on. Also #60977. It's not a proposal in its current form as that needs a more holistic analysis.

Right now there are some characters permitted to be used in subtest names that might cause problems for other systems that need to process go test -json output, including the Unicode replacement character U+FFFD:

package main

import (
	"strconv"
	"testing"
)

func Test(t *testing.T) {
	const r = '\uFFFD' // https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character
	t.Run(string(r), func(t *testing.T) {
		t.Logf("Hello, %U.", r)
		if strconv.IsPrint(r) {
			t.Logf("%q is a printable rune.", r)
		}
	})

	// Output:
	// === RUN   Test
	// === RUN   Test/�
	//     prog_test.go:11: Hello, U+FFFD.
	//     prog_test.go:13: '�' is a printable rune.
	// --- PASS: Test (0.00s)
	//     --- PASS: Test/� (0.00s)
	// PASS
}

(playground link: https://go.dev/play/p/unQcgsua0J3)

Also potentially worth considering are subtest names that use different Unicode normalization, which the current basic rewriting of equal subtest names doesn't catch: https://go.dev/play/p/xut7W4Cp4p_g.

Said programs can (and do) deal with this by escaping on their end, but this issue is to investigate whether a future version of the go command can do better, perhaps by disallowing or rewriting subtest names.

Note that a subtest name is often used in combination with other parts needed to uniquely identify a test, which have corresponding existing constraints:

• import path (https://pkg.go.dev/golang.org/x/mod/module#CheckImportPath, https://go.dev/ref/mod#go-mod-file-ident, may expand to include Unicode in proposal: cmd/go: allow unicode in module paths and file names #67562)
• identifier name (https://go.dev/ref/spec#Identifiers)
• subtest name (https://cs.opensource.google/go/go/+/master:src/testing/match.go;l=282-284;drc=3bc28402fae2a1646e4d2756344b5eb34994d25f)

CC @matloob, @samthanawalla.

[patrickmeiring]

A further point: go subtest names are separated by forward slashes in go test output ('/'), but these separators cannot be reliably identified by downstream systems as the subtest names passed t.Run(...) are allowed to contain a slash (and these slashes are not escaped or encoded). This makes it error prone for tools to interpret the hierarchy of subtest results.

[matloob]

Are you suggesting that we fail subtests that have those characters in their names or that we sanitize the names?

[patrickmeiring]

@matloob Initially my thought was that any uses of them could be unambiguously encoded (e.g. all uses of '' and '/' escaped with a '') as this limits migration difficulty, but barring their use is also an option.

For folks using the results programatically, having the components of the test ID separately reported in the structured output format
(https://pkg.go.dev/cmd/test2json#hdr-Output_Format) (however internally represented), i.e. as a []string, would be a nice addition but I am digressing slightly from the focus of this bug.