Make il8n tools available from `python -m`

[bbkane]

BPO	36837
Nosy	@warsaw, @abadger, @bbkane

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-07.16:23:26.059>
labels = []
title = 'Make il8n tools available from `python -m`'
updated_at = <Date 2019-05-11.15:26:52.551>
user = 'https://github.com/bbkane'

bugs.python.org fields:

activity = <Date 2019-05-11.15:26:52.551>
actor = 'a.badger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2019-05-07.16:23:26.059>
creator = 'bbkane'
dependencies = []
files = []
hgrepos = []
issue_num = 36837
keywords = []
message_count = 5.0
messages = ['341771', '341876', '341999', '342005', '342198']
nosy_count = 3.0
nosy_names = ['barry', 'a.badger', 'bbkane']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue36837'
versions = []

[bbkane]

Localizing a Python application involves using the gettext standard library module to read .mo files. There are three scripts to assist with this in https://github.com/bbkane/cpython/tree/master/Tools/i18n :

• makelocalealias.py : Convert the X11 locale.alias file into a mapping dictionary suitable for locale.py.
• msgfmt.py : Generate binary message catalog from textual translation description
• pygettext.py : Generate .pot files identical to what GNU xgettext[2]
  generates for C and C++ code (these can be translated by msgfmt.py)

I recently wrote a tutorial to localize a Python Script ( https://github.com/bbkane/arcade/blob/bbkane/add_localization_example/doc/examples/text_loc_example.rst ) and I had to tell my users (a student audience) to download these scripts from GitHub. I would have been much happier to ask them to use a builtin Python tool available from the -m switch (similar to python -m json.tool), so this issue is to add that.

The docs ( https://docs.python.org/3/library/gettext.html#internationalizing-your-programs-and-modules ) mention these scripts, but do not provide any information on how to get them.

Possible solutions:

• turn gettext.py into a package and put these scripts into a tool subpackage (similar to json.tool)
• Add a separate package (il8n perhaps) and put these scripts into there
• Add links to these scripts and instructions to use them in the docs.

[warsaw]

One other suggestion: put the bulk of Tools/i18n/pygettext.py into Lib/_pygettext.py, then import its main() in both Lib/gettext.py and Tools/i18n/pygettext.py. Then just call that main().

[abadger]

Note, I've been doing some tests of how our gettext module differs from GNU gettext and run into a few bugs and lack of features which make msgfmt unusable and limit pygettext's usefulness.

• msgfmt doesn't seem to store the charset from the .po file into the .mo file. I think this might have been okay for the lgettext() and gettext() methods under Python2 as those probably passed the byte strings from the .mo files through verbatim. Under Python3, however, we have to decode the byte strings to text and we can't do that without knowing the charset. This leads to a UnicodeDecodeError on any .mo file which contains non-ascii characters (which is going to be the majority of them)

• So far, I have found that pygettext doesn't understand how to extract strings from ngettext(). This means that your code can't use plural forms if you want to use pygettext to extract the strings.

These deficiencies are probably things that need to be fixed if we're going to continue to promote these tools in the documentation.

[abadger]

A note about the msgfmt problem. It looks like GNU gettext's msgfmt has a similar problem but the msgfmt from pybabel does not. This may mean that we need to change the gettext *Translation objects to be more tolerant of non-ascii encodings (perhaps defaulting to utf-8 instead of ascii).

[abadger]

Scratch what I said in https://bugs.python.org/issue36837?@ok_message=msg%20342005%20created%0Aissue%2036837%20message_count%2C%20messages%20edited%20ok&@template=item#msg342005

GNU msgfmt does extract the charset correctly. (My previous test failed to write any output so it was using the .mo file I had written out with msgfmt.py. I realized that this morning when I figured out why my C test program wasn't finding any message catalog.

For reference the three ways to extract strings with the three tools are:

• pygettext.py test.py
• pybabel extract -o messages.pot test.py
• xgettext test.py -o messages.pot test.py

and the three ways to generate catalogs via the three tools are:

• msgfmt3.7.py es_MX/LC_MESSAGES/domain.po
• msgfmt es_MX/LC_MESSAGES/testc.po -o es_MX/LC_MESSAGES/testc.mo
• pybabel compile -D test -d . [--use-fuzzy]

[StanFromIreland]

I think @warsaw 's suggestion is quite good, I can open a PR if there is some agreement that is the way to go.

cc @tomasr8 Since you like working on pygettext :-)

[tomasr8]

Yeah I was going to re-propose this once I have fixed most of the issues w/ pygettext and msgfmt. I've spent quite a long time on just fixing pygettext (didn't even get to msgfmt yet 😆 ) and I had this in mind as the final step once the tools are ready to be exposed for a wider use (currently they're somewhat hidden inside Tools which is not even available on most distributions IIRC).

If you don't mind, I'd like to do this part myself. However, there's still a couple of things I'd like to implement in pygettext and msgfmt. If you'd like to help I'm happy to split them with you 🙂

[StanFromIreland]

Sure what are you planning?

[tomasr8]

Couple things :)

• Deprecate or remove the extractall option (currently a noop)
• Possibly add --omit-header (would also be useful for some of our tests as it would allows us to switch to a full POT in lib/test/translationdata)
• width not being respected for msgids but only for location comments
• Some weird formatting for strings containing format specifiers
• Cleaning up some obsolete tests from test_i18n
• and a couple more things which I still need to look into

Let me know which one you'd like to take and I can explain in more detail

[StanFromIreland]

I'll look through the 35 tests in test_i18n.

[StanFromIreland]

I was unable to find any tests I would consider obsolete, maybe I missed something? There is some duplication however. test_funcdocstring_multiple_funcs covers three of the previous cases.