Lingua 4.x breaks the Lingua extractor

[sqlalchemy-bot]

Migrated issue, originally created by Laurent DAVERIO (@ldaverio)

When upgrading from Lingua 3.12 to Lingua 4.x, an attempt to extract strings from a Mako template fails with the following message:

File "/Users/xxxxxxx/pydev/lib/python3.5/tokenize.py", line 560, in _tokenize
if line[pos] in '#\r\n': # skip comments or blank lines

This message also appeared in issue #72 of the wichert/lingua repository on GitHub:

wichert/lingua#72

The problem seems to be that "tokenize requires unicode input". And in effect, it seems that I can get rid of the problem by changing line 35 of ext/linguaplugin.py from :

code = io.ByteIO(source)

to:

code = io.StringIO(str(source, encoding='utf-8'))

I'm not sure of all the implications yet, so I don't want to issue a pull request, but this fix seems to work for me.

[sqlalchemy-bot]

Changes by Laurent DAVERIO (@ldaverio):

• edited description

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

this is all @wichert. Perhaps he has some ideas?

[sqlalchemy-bot]

Wichert Akkerman (@wichert) wrote:

This is my fault: I had not realised that a recent refactoring impacted the API used by plugins. Since this is likely to impact more plugins I think this should be addressed in lingua.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

• cap lingua at <4 for the moment, references Lingua 4.x breaks the Lingua extractor #255

→ f9faef9

[sqlalchemy-bot]

Laurent DAVERIO (@ldaverio) wrote:

FWIW, I've been using the patched version of Mako (with the change in
ext/linguaplugin.py) since I reported the problem, everything works fine
for me :)

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

that's not fixing the failure I'm getting, which is:

#!

========================================================================= FAILURES =========================================================================
_____________________________________________________________ ExtractMakoTestCase.test_extract _____________________________________________________________
Traceback (most recent call last):
  File "/home/classic/dev/mako/test/ext/test_linguaplugin.py", line 30, in test_extract
    plugin(os.path.join(template_base, 'gettext.mako'), MockOptions()))
  File "/home/classic/dev/mako/mako/ext/extract.py", line 13, in process_file
    for extracted in self.extract_nodes(template_node.get_children()):
  File "/home/classic/dev/mako/mako/ext/extract.py", line 91, in extract_nodes
    code, node.lineno - 1, translator_strings):
  File "/home/classic/dev/mako/mako/ext/linguaplugin.py", line 36, in process_python
    self.filename, self.options, code, code_lineno -1):
  File "/home/classic/dev/mako/.tox/py27/lib/python2.7/site-packages/lingua/extractors/python.py", line 356, in __call__
    return parser(token_stream, options, filename, lineno)
  File "/home/classic/dev/mako/.tox/py27/lib/python2.7/site-packages/lingua/extractors/python.py", line 153, in __call__
    if options.comment_tag is True:
AttributeError: MockOptions instance has no attribute 'comment_tag'
===================================================== 1 failed, 460 passed, 2 skipped in 11.15 seconds =====================================================
ERROR: InvocationError: '/home/classic/dev/mako/.tox/py27/bin/py.test'
_________________________________________________________________________ summary __________________________________________________________________________
ERROR:   py27: commands failed

this might be just local to the test but I don't have resources to debug this plugin myself.

[sqlalchemy-bot]

Wichert Akkerman (@wichert) wrote:

I've pushed a change to lingua which allows passing bytes instances to the Python extractor again. @ldaverio can you please check if that fixes your problem?

[sqlalchemy-bot]

Laurent DAVERIO (@ldaverio) wrote:

Hello Wichert, sorry for the late reply.

I've just tested the 4 following combinations :

• 2 Mako versions (1.0.4, and my git)
• 2 Lingua versions (4.8, and 4.9)

With Lingua 4.8, my patched Mako works for me, the other fails with:

Traceback (most recent call last):
  File "/usr/home/daverio/pydev/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 280, in main
    for message in extractor(real_filename, options):
  File "/usr/home/daverio/pydev/mako/mako/ext/extract.py", line 13, in process_file
    for extracted in self.extract_nodes(template_node.get_children()):
  File "/usr/home/daverio/pydev/mako/mako/ext/extract.py", line 91, in extract_nodes
    code, node.lineno - 1, translator_strings):
  File "/usr/home/daverio/pydev/mako/mako/ext/linguaplugin.py", line 36, in process_python
    self.filename, self.options, code, code_lineno -1):
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 356, in __call__
    return parser(token_stream, options, filename, lineno)
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 166, in __call__
    for (token_type, token, location, _) in token_stream:
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 140, in next
    token = self._transform(next(self.queue))
  File "/usr/home/daverio/pydev/lib/python3.5/tokenize.py", line 560, in _tokenize
    if line[pos] in '#\r\n':           # skip comments or blank lines
TypeError: 'in <string>' requires string as left operand, not int

With Lingua 4.9, both fail with a second error message (but the UnicodeWarning message happens only with regular Mako 1.0.4):

/usr/home/daverio/pydev/lib/python3.5/site-packages/mako/ext/linguaplugin.py:36: UnicodeWarning: Python extractor called with bytes input. Please update your plugin to submit unicode instead.
  self.filename, self.options, code, code_lineno -1):
Traceback (most recent call last):
  File "/usr/home/daverio/pydev/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 356, in main
    save_catalog(catalog, options.output)
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 230, in save_catalog
    if old_catalog is not None and identical(catalog, old_catalog):
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 220, in identical
    return a == b
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 64, in __eq__
    r = super(POEntry, self).__eq__(other)
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/polib-1.0.7-py3.5.egg/polib.py", line 1083, in __eq__
  File "/usr/home/daverio/pydev/lib/python3.5/site-packages/polib-1.0.7-py3.5.egg/polib.py", line 1059, in __cmp__
TypeError: string indices must be integers

I can't investigate the problem right now, but I'll try to do it asap, possibly this weekend.

[sqlalchemy-bot]

Laurent DAVERIO (@ldaverio) wrote:

Hello again, in addition to the Lingua 4.9 problem, which forced me to stay at 4.8.1, it now appears that setuptools >= 20.6.6 breaks "po-create" :( :( :(

So, I have to pin setuptools==20.4...

Traceback (most recent call last):
  File "/Users/daverio/pydev/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/Users/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 252, in main
    register_babel_plugins()
  File "/Users/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
    extractor = entry_point.load(require=True)
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2206, in load
    self.require(*args, **kwargs)
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2223, in require
    items = working_set.resolve(reqs, env, installer)
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 814, in resolve
    if req.marker and not req.marker.evaluate():
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.