Skip to content

gh-126700: pygettext: Support more gettext functions #126912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Nov 22, 2024
Merged

gh-126700: pygettext: Support more gettext functions #126912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
22f7803
Support multi-argument gettext functions
tomasr8 Nov 16, 2024
06686c9
Update snapshots
tomasr8 Nov 16, 2024
e5c2746
Bump pygettext version
tomasr8 Nov 16, 2024
511a0c0
Add news entry
tomasr8 Nov 16, 2024
62d6455
Correctly count enclosures
tomasr8 Nov 17, 2024
496f5d9
Restore warnings for invalid arguments
tomasr8 Nov 17, 2024
06186a0
Remove extra space
tomasr8 Nov 17, 2024
3d67a7a
Simplify code
tomasr8 Nov 17, 2024
48070d5
Update comment
tomasr8 Nov 17, 2024
d6fd789
Only extract when __enclosure_count is 0
tomasr8 Nov 18, 2024
3497690
Keep the old version
tomasr8 Nov 18, 2024
192187e
Improve news entry
tomasr8 Nov 18, 2024
9cfc901
Update snapshots
tomasr8 Nov 18, 2024
24851c5
Refactor __openseen
tomasr8 Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 34 additions & 12 deletions Lib/test/test_tools/i18n_data/messages.pot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,53 +15,75 @@ msgstr ""
"Generated-By: pygettext.py 1.5\n"


#: messages.py:5
#: messages.py:16
msgid ""
msgstr ""

#: messages.py:8 messages.py:9
#: messages.py:19 messages.py:20
msgid "parentheses"
msgstr ""

#: messages.py:12
#: messages.py:23
msgid "Hello, world!"
msgstr ""

#: messages.py:15
#: messages.py:26
msgid ""
"Hello,\n"
" multiline!\n"
msgstr ""

#: messages.py:29
#: messages.py:46 messages.py:89 messages.py:90 messages.py:93 messages.py:94
#: messages.py:99
msgid "foo"
msgid_plural "foos"
msgstr[0] ""
msgstr[1] ""

#: messages.py:47
msgid "something"
msgstr ""

#: messages.py:50
msgid "Hello, {}!"
msgstr ""

#: messages.py:33
#: messages.py:54
msgid "1"
msgstr ""

#: messages.py:33
#: messages.py:54
msgid "2"
msgstr ""

#: messages.py:34 messages.py:35
#: messages.py:55 messages.py:56
msgid "A"
msgstr ""

#: messages.py:34 messages.py:35
#: messages.py:55 messages.py:56
msgid "B"
msgstr ""

#: messages.py:36
#: messages.py:57
msgid "set"
msgstr ""

#: messages.py:42
#: messages.py:63
msgid "nested string"
msgstr ""

#: messages.py:47
#: messages.py:68
msgid "baz"
msgstr ""

#: messages.py:91 messages.py:92 messages.py:95 messages.py:96
msgctxt "context"
msgid "foo"
msgid_plural "foos"
msgstr[0] ""
msgstr[1] ""

#: messages.py:100
msgid "domain foo"
msgstr ""

52 changes: 49 additions & 3 deletions Lib/test/test_tools/i18n_data/messages.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Test message extraction
from gettext import gettext as _
from gettext import (
gettext,
ngettext,
pgettext,
npgettext,
dgettext,
dngettext,
dpgettext,
dnpgettext
)

_ = gettext

# Empty string
_("")
Expand All @@ -21,13 +32,23 @@
_(None)
_(1)
_(False)
_(x="kwargs are not allowed")
_(("invalid"))
_(["invalid"])
_({"invalid"})
_("string"[3])
_("string"[:3])
_({"string": "foo"})

# pygettext does not allow keyword arguments, but both xgettext and pybabel do
_(x="kwargs work!")

# Unusual, but valid arguments
_("foo", "bar")
_("something", x="something else")

# .format()
_("Hello, {}!").format("world") # valid
_("Hello, {}!".format("world")) # invalid
_("Hello, {}!".format("world")) # invalid, but xgettext and pybabel extract the first string

# Nested structures
_("1"), _("2")
Expand Down Expand Up @@ -62,3 +83,28 @@ def _(x):

def _(x="don't extract me"):
pass


# Other gettext functions
gettext("foo")
ngettext("foo", "foos", 1)
pgettext("context", "foo")
npgettext("context", "foo", "foos", 1)
dgettext("domain", "foo")
dngettext("domain", "foo", "foos", 1)
dpgettext("domain", "context", "foo")
dnpgettext("domain", "context", "foo", "foos", 1)

# Complex arguments
ngettext("foo", "foos", 42 + (10 - 20))
dgettext(["some", {"complex"}, ("argument",)], "domain foo")

# Invalid calls which are not extracted
gettext()
ngettext('foo')
pgettext('context')
npgettext('context', 'foo')
dgettext('domain')
dngettext('domain', 'foo')
dpgettext('domain', 'context')
dnpgettext('domain', 'context', 'foo')
4 changes: 2 additions & 2 deletions Lib/test/test_tools/test_i18n.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,14 +332,14 @@ def test_calls_in_fstring_with_multiple_args(self):
msgids = self.extract_docstrings_from_str(dedent('''\
f"{_('foo', 'bar')}"
'''))
self.assertNotIn('foo', msgids)
self.assertIn('foo', msgids)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both xgettext and pybabel extract this and it would take a lot of effort to disallow this in general with the current extractor, so I'd leave this for now at least.

self.assertNotIn('bar', msgids)

def test_calls_in_fstring_with_keyword_args(self):
msgids = self.extract_docstrings_from_str(dedent('''\
f"{_('foo', bar='baz')}"
'''))
self.assertNotIn('foo', msgids)
self.assertIn('foo', msgids)
self.assertNotIn('bar', msgids)
self.assertNotIn('baz', msgids)

Expand Down
2 changes: 2 additions & 0 deletions Lib/test/translationdata/argparse/msgids.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ argument %(argument_name)s: %(message)s
argument '%(argument_name)s' is deprecated
can't open '%(filename)s': %(error)s
command '%(parser_name)s' is deprecated
conflicting option string: %s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should msgid_plural also be output? Or do this in the following PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also want to include msgctxt even though I don't think any messages use pgettext currently. I was thinking instead of using a list of msgids, why not use the generated POT file?

We initially rejected that idea when adding the snapshots because of potentially changing line locations, but pygettext has an option to turn those off.

We could even add an option to not emit the header (pybabel has this for instance). Then the snapshots would only change if the strings themselves change.

expected %s argument
expected at least one argument
expected at most one argument
expected one argument
Expand Down
1 change: 1 addition & 0 deletions Lib/test/translationdata/optparse/msgids.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
%(option)s option requires %(number)d argument
%prog [options]
%s option does not take a value
Options
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Tools-Demos/2024-11-16-20-47-20.gh-issue-126700.ayrHv4.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support for multi-argument :mod:`gettext` functions in :program:`pygettext.py`.
Loading
Loading