Improved javascript template string expression extracting #792

gitaarik · 2021-06-09T11:43:01Z

Add option parse_template_string that improves parsing of javascript templates expressions, like:

`${gettext('my text')}`

And it also works with nested expressions like:

${`some text ${gettext('my text')} more text`}

You have to enable template string extraction with an option in the option_map:

extract_from_dir(
    # ...
    options_map={
        '**': {'parse_template_string': True}
    }
)

Then it should work.

gitaarik · 2021-06-09T11:43:32Z

Hi @akx . I've been working on this feature for a while. In my setup I need these changes. However I have some problems when using my own fork in my project. So I have some questions regarding using a fork of babel in my project, and also what needs to be done to get this merged, so I don't have to use my fork anymore.

In my project, which uses Pipenv. Inside the Pipfile, I use a VCS descriptor to specify my python-babel fork:

babel = {editable = true, ref = "javascript-improvements-1", git = "https://github.com/gitaarik/babel"}

This works partly. When I run python-babel's babel.messages.extract.extract_from_dir, it finds gettext() functions that it didn't found before.

However, when I run my Django devserver, on some requests (not all of them, I'm not sure yet why it only happens on some requests), the request fails with a exception coming from python-babel:

RuntimeError: The babel data files are not available. This usually happens because you are using a source checkout from Babel and you did not build the data files.  Just make sure to run "python setup.py import_cldr" before installing the library.

I noticed that when I go into the virtualenv directory (which is created by Pipenv and can be retrieved with pipenv --venv) and then from there go to src/babel/, I can run the command python setup.py import_cld to generate the locale files and then it seems to work.

I will have to look into the possibilities to integrate this in my project in a way that the locale files would be generated when I build and deploy my project. In the meantime it would be good for us if the changes would eventually be merged into the official package.

So I have two questions:

Are there easier ways of using my fork in my project, for development and deployment?
What would be needed to get these changes merged into the official package, so that we don't have to make things complicated with a fork?

akx · 2022-01-28T15:05:43Z

Hi @gitaarik, sorry for the latency. I had somehow inadvertently turned all notifications about Babel off (or maybe had just dropped the ball on this one).

Could you rebase this on master and maybe add a test that shows how this works?

akx

Beyond the other comments, I'd appreciate some tests :)

akx · 2022-01-28T15:07:52Z

babel/messages/extract.py

+
+                    fake_file_obj = io.BytesIO(expression_contents.encode())
+
+                    for item in extract_javascript(fake_file_obj, keywords, comment_tags, options):


How does this interact with e.g. line numbering?

line numbering has been fixed in #939. There is now also a test for that.

akx · 2022-01-28T15:07:54Z

babel/messages/extract.py

+def parse_template_string(template_string, fileobj, keywords, comment_tags, options):
+
+    prev_character = None
+    level = 0
+    inside_str = False
+    expression_contents = ''
+
+    for character in template_string[1:-1]:
+
+        if not inside_str and character in ('"', "'", '`'):
+            inside_str = character
+        elif inside_str == character and prev_character != r'\\':
+            inside_str = False
+
+        if level:
+            expression_contents += character
+
+        if not inside_str:
+
+            if character == '{' and prev_character == '$':
+                level += 1
+
+            elif level and character == '}':
+
+                level -= 1
+
+                if level == 0 and expression_contents:
+
+                    expression_contents = expression_contents[0:-1]
+
+                    fake_file_obj = io.BytesIO(expression_contents.encode())
+
+                    for item in extract_javascript(fake_file_obj, keywords, comment_tags, options):
+                        yield item
+
+                    expression_contents = ''
+
+        prev_character = character


Could you remove the empty lines within the blocks, so the spacing of this code (which is quite airy 😄 ) matches the rest of the codebase?

Done in #939.

johanneswilm · 2022-12-29T14:23:00Z

Beyond the other comments, I'd appreciate some tests :)

Tests have been added in #939.

Improved javascript template string expression extracting

28e7b15

This was referenced Jun 9, 2021

Javascript lexer improvements #773

Closed

Template strings support in JS lexer #329

Open

akx requested changes Jan 28, 2022

View reviewed changes

akx added bug enhancement area/messages labels Jan 28, 2022

johanneswilm mentioned this pull request Dec 29, 2022

javascript extract improvements #939

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved javascript template string expression extracting #792

Improved javascript template string expression extracting #792

gitaarik commented Jun 9, 2021

gitaarik commented Jun 9, 2021 •

edited

akx commented Jan 28, 2022

akx left a comment

akx Jan 28, 2022

johanneswilm Dec 29, 2022

akx Jan 28, 2022

johanneswilm Dec 29, 2022

johanneswilm commented Dec 29, 2022


		fake_file_obj = io.BytesIO(expression_contents.encode())

		for item in extract_javascript(fake_file_obj, keywords, comment_tags, options):

Improved javascript template string expression extracting #792

Are you sure you want to change the base?

Improved javascript template string expression extracting #792

Conversation

gitaarik commented Jun 9, 2021

gitaarik commented Jun 9, 2021 • edited

akx commented Jan 28, 2022

akx left a comment

Choose a reason for hiding this comment

akx Jan 28, 2022

Choose a reason for hiding this comment

johanneswilm Dec 29, 2022

Choose a reason for hiding this comment

akx Jan 28, 2022

Choose a reason for hiding this comment

johanneswilm Dec 29, 2022

Choose a reason for hiding this comment

johanneswilm commented Dec 29, 2022

gitaarik commented Jun 9, 2021 •

edited