Implement localization using gettext files — I18N — L10N #2090

Goutte · 2023-12-14T17:14:40Z

See #1134

Features

Recipe to extract new translations from the Go code: make i18n_extract
Using widely supported gettext format
Embedded lightweight MO files
Detect language from environment variables
Some strings were pluralized

Issues

No regional fallback for now, waiting for leonelquinteros/gotext#85 to be merged.

This will also unlock fallback to english when a translation is not found.

Replaces #1944

github-actions · 2023-12-14T17:14:59Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-12-15T04:30:17Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

- Recipe to extract new translations from the Go code: `make i18n_extract` - Embedded `MO` files - Detect language from environment variables - Some strings were pluralized

github-actions · 2023-12-16T09:56:20Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Goutte · 2023-12-16T10:04:57Z

I know it's a lot of added lines, but more than half are from the POT and PO files. I added the french translation as an example, for good measure.

marckhouzam · 2023-12-22T19:13:37Z

Thanks @Goutte!

A couple of things to help get this reviewed:
1- can resolve the conflicts with main
2- can you add in the description how to test this (including the best way to install xgotext)

marckhouzam · 2023-12-22T20:52:12Z

@Goutte could you explain what the process would be, if we adopted this PR, for a cobra contributor to add a new string printout? It is not clear to me when make i18n_extract should be run and which *.po/default.pot/*.mo files need to be modified by hand and when, etc.

At this point, clarifying these steps in the PR description is sufficient, but if the review goes well, we'll need those details in some cobra documentation.

Also, I'm also interested in understanding how a project using Cobra would proceed to add translations, or is that completely up to the project and this PR does not help?

marckhouzam · 2023-12-22T21:03:30Z

@Goutte could you explain what the process would be, if we adopted this PR, for a cobra contributor to add a new string printout? It is not clear to me when make i18n_extract should be run and which *.po/default.pot/*.mo files need to be modified by hand and when, etc.

I see this already in the docs. Nice work.
Can you clarify what you mean by "Make sure your software has also updated the MO files"?
Do we absolutely need to use a special editor like poedit or can this be done from the command-line?

marckhouzam

Just started looking at this but I'm staring to get it.
Seems good at first impression.

But, I feel we absolutely need a way to generate the *.mo files from the command-line, through the makefile in fact.

Makefile

marckhouzam · 2023-12-22T19:37:36Z

Makefile

@@ -33,3 +33,7 @@ install_deps:

 clean:
 	rm -rf $(BIN)
+
+i18n_extract:


Do I understand correctly that whenever a file changes, this make target must be re-run to update the line numbers in the default.pot file?

If that is the case, then it is essential that we run this make target in .github/test.yml and check if default.pot (or anything else) gets changed. I don't expect many contributors to know they have to run this, so CI should fail if they don't

localizer.go

marckhouzam · 2023-12-22T21:19:27Z

args.go

@@ -33,15 +34,15 @@ func legacyArgs(cmd *Command, args []string) error {

 	// root command with subcommands, do subcommand checking.
 	if !cmd.HasParent() && len(args) > 0 {
-		return fmt.Errorf("unknown command %q for %q%s", args[0], cmd.CommandPath(), cmd.findSuggestions(args[0]))
+		return fmt.Errorf(gotext.Get("LegacyArgsValidationError"), args[0], cmd.CommandPath(), cmd.findSuggestions(args[0]))


How about leaving the original english string as the index for all the gotext.Get() calls?
It would make reading the code much easier and it would make working with the *.po files easier as the original text would be right there as the index.

If the % formatting gives a problem, maybe we can replace it with %% in the index? It is not ideal, but it would be manageable. So, for this line here we would instead use (unless there is a better way?)

return fmt.Errorf(gotext.Get("unknown command %%q for %%q%%s"), args[0], cmd.CommandPath(), cmd.findSuggestions(args[0]))

I’ve looked at gotext in more detail, and the % form will work just fine as long as the parameters are in the gotext.Get() call instead of outside.

This is the crux of the matter.

I've done a whole bunch of implementations of translations in the past, and every time I try to use plain english instead of keys I end up either regretting it or refactoring heavily to keys.

Here's some food for thought when using raw english as keys:

cosmetically changing the english string (typo, whitespace, etc.) invalidates all the existing translations (possibly hundreds of files need to be updated, for nothing)

compels usage of context shenanigans for strings that are the same in english in different contexts but different in other languages

too many gettext parsers out there choke on nasty translation ids like the "quoted q'tara" <tag>

sometimes it creates a bias towards english in the structure of strings, that need to be fixed by translators, straight in the code, once again invalidating all existing translations

I'm very aware that using keys makes the code harder to read and understand. This is mitigated a little by a careful choice of the wording of the key.

I had to pick one way or the other ; it was not an easy choice, but it's one I made many times and I decided to go with hindsight from past experiences.

I'm not adamant on this, quite the contrary.
I listed some of the key points of my decision above (probably forgot some) ; I'll let y'all be the final judges. Good luck !

marckhouzam · 2023-12-22T21:22:37Z

args.go

@@ -16,6 +16,7 @@ package cobra

 import (
 	"fmt"
+	"github.com/leonelquinteros/gotext"


We could import this as i18n, so that instead of calling gotext.Get() we would call i18n.Get().
This may be clearer when browsing the code?

marckhouzam · 2023-12-23T01:52:17Z

Something to discuss later, I just don’t want to forget.

We will need a way for a program using cobra to disable these translations. For example, if a mycli project does not translate its own strings, it might not want the cobra strings to get translated as it would give an inconsistent user experience.

So we probably want the project to explicitly have to turn on translations through a global setting.

Goutte · 2023-12-23T13:15:02Z

Thanks for the comprehensive review, @marckhouzam !

Also, I'm also interested in understanding how a project using Cobra would proceed to add translations, or is that completely up to the project and this PR does not help?

This PR does not help devs to implement their I18N.
They are free to choose whatever solution they like.

We could perhaps provide some doc/sugar on how to use the same translation toolkit as cobra, since the relevant libraries are already included ?

Can you clarify what you mean by "Make sure your software has also updated the MO files"? Do we absolutely need to use a special editor like poedit or can this be done from the command-line?

Good question. Since gettext is mature, I'm pretty confident there are CLI tools out there handling this. (or we'll have to make one, preferably in Golang, using cobra of course)

I'd like the MO to be generated by CI as well, if only to prevent contributors to inject nasty things in the binary files.

I'll look into this.

So we probably want the project to explicitly have turn on translations through a global setting.

Very good point. Partially translated apps are awkward, which is precisely why I wanted cobra to support i18n. I'm not sure right now how to proceed to add a global setting to cobra, but I'll look into it and come back to ask for help if I don't find how. Do give tips if you'd prefer it to be done in some specific way :)

marckhouzam · 2023-12-23T14:07:52Z

To allow a project to enable translations I’m thinking of two approaches you can take:

A global setting like cobra has for other features:

cobra/cobra.go

Lines 52 to 66 in 4122785

    
           // EnablePrefixMatching allows setting automatic prefix matching. Automatic prefix matching can be a dangerous thing 
        
           // to automatically enable in CLI tools. 
        
           // Set this to true to enable it. 
        
           var EnablePrefixMatching = defaultPrefixMatching 
        
           // EnableCommandSorting controls sorting of the slice of commands, which is turned on by default. 
        
           // To disable sorting, set it to false. 
        
           var EnableCommandSorting = defaultCommandSorting 
        
           // EnableCaseInsensitive allows case-insensitive commands names. (case sensitive by default) 
        
           var EnableCaseInsensitive = defaultCaseInsensitive 
        
           // EnableTraverseRunHooks executes persistent pre-run and post-run hooks from all parents. 
        
           // By default this is disabled, which means only the first run hook to be found is executed. 
        
           var EnableTraverseRunHooks = defaultTraverseRunHooks

A field in the root command like we have for tuning shell completions

cobra/command.go

Line 203 in 4122785

CompletionOptions CompletionOptions

if you think different settings might be useful eventually, I would recommend using a struct so we could grow it overtime

marckhouzam · 2023-12-23T14:10:44Z

This PR does not help devs to implement their I18N. They are free to choose whatever solution they like.

We could perhaps provide some doc/sugar on how to use the same translation toolkit as cobra, since the relevant libraries are already included ?

Thanks for clarifying, it makes sense. Let’s leave this for a potential separate PR and focus on cobra’s strings .

Luap99 · 2023-12-28T14:28:11Z

I think cobra should have a built tag to disable/enable translations. Right now it looks like all translations are embedded in the final binary. That will lead to unnecessary bloat for users who do not want to use it so a build tag would be the better option IMO.

And these translations should be opt in not opt out IMO.

marckhouzam · 2023-12-28T20:32:17Z

I think cobra should have a built tag to disable/enable translations IMO.

Good idea +1

And these translations should be opt in not opt out IMO.

Agreed 👍

github-actions · 2024-01-10T15:08:20Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

…cies

github-actions · 2024-01-10T15:14:07Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Goutte · 2024-01-10T15:24:30Z

I think cobra should have a built tag to disable/enable translations. Right now it looks like all translations are embedded in the final binary. That will lead to unnecessary bloat for users who do not want to use it so a build tag would be the better option IMO.

And these translations should be opt in not opt out IMO.

That's a good idea.

I think I managed to do this using the "locales" build tag.

Now I need to tweak the test suite to use that build tag,
or perhaps even make two runs, with and without.

Anyhow, for now the tests are broken until I figure this out.

github-actions · 2024-01-10T15:45:15Z

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Goutte mentioned this pull request Dec 14, 2023

Multi-language support for CLIs? #1134

Open

Goutte force-pushed the feat-i18n-gettext branch from caedfc5 to fb482c2 Compare December 15, 2023 04:30

Goutte mentioned this pull request Dec 16, 2023

Add support for multiple languages leonelquinteros/gotext#85

Merged

3 tasks

feat(i18n): implement localization using gettext files

ab14efa

- Recipe to extract new translations from the Go code: `make i18n_extract` - Embedded `MO` files - Detect language from environment variables - Some strings were pluralized

Goutte force-pushed the feat-i18n-gettext branch from fb482c2 to ab14efa Compare December 16, 2023 09:56

marckhouzam added kind/feature A feature request for cobra; new or enhanced behavior area/cobra-command Core `cobra.Command` implementations labels Dec 22, 2023

marckhouzam mentioned this pull request Dec 22, 2023

Internationalization (I18N) #1944

Open

marckhouzam reviewed Dec 22, 2023

View reviewed changes

feat(i18n): only embed locale files when using the build tag 'locales'

c922911

feat(i18n): add a Makefile recipe to install i18n extraction dependen…

20e0d82

…cies

Goutte added 2 commits January 10, 2024 16:37

chore(i18n): lint

f7f98c4

feat(i18n): test the locales using the appropriate build flag

d362950

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement localization using gettext files — I18N — L10N #2090

Implement localization using gettext files — I18N — L10N #2090

Goutte commented Dec 14, 2023 •

edited

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 16, 2023

Goutte commented Dec 16, 2023

marckhouzam commented Dec 22, 2023

marckhouzam commented Dec 22, 2023

marckhouzam commented Dec 22, 2023

marckhouzam left a comment

marckhouzam Dec 22, 2023

marckhouzam Dec 22, 2023 •

edited

marckhouzam Dec 23, 2023 •

edited

Goutte Dec 23, 2023

marckhouzam Dec 22, 2023

marckhouzam commented Dec 23, 2023 •

edited

Goutte commented Dec 23, 2023

marckhouzam commented Dec 23, 2023

marckhouzam commented Dec 23, 2023

Luap99 commented Dec 28, 2023

marckhouzam commented Dec 28, 2023

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

Goutte commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

Implement localization using gettext files — I18N — L10N #2090

Are you sure you want to change the base?

Implement localization using gettext files — I18N — L10N #2090

Conversation

Goutte commented Dec 14, 2023 • edited

Features

Issues

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 16, 2023

Goutte commented Dec 16, 2023

marckhouzam commented Dec 22, 2023

marckhouzam commented Dec 22, 2023

marckhouzam commented Dec 22, 2023

marckhouzam left a comment

Choose a reason for hiding this comment

marckhouzam Dec 22, 2023

Choose a reason for hiding this comment

marckhouzam Dec 22, 2023 • edited

Choose a reason for hiding this comment

marckhouzam Dec 23, 2023 • edited

Choose a reason for hiding this comment

Goutte Dec 23, 2023

Choose a reason for hiding this comment

marckhouzam Dec 22, 2023

Choose a reason for hiding this comment

marckhouzam commented Dec 23, 2023 • edited

Goutte commented Dec 23, 2023

marckhouzam commented Dec 23, 2023

marckhouzam commented Dec 23, 2023

Luap99 commented Dec 28, 2023

marckhouzam commented Dec 28, 2023

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

Goutte commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

Goutte commented Dec 14, 2023 •

edited

marckhouzam Dec 22, 2023 •

edited

marckhouzam Dec 23, 2023 •

edited

marckhouzam commented Dec 23, 2023 •

edited