Translations and underscore locales

Overview

We use Transifex and the gettext gem for translations (with fast_gettext on top to speed things up, and Globalize for the PublicBody model translations). There's a good background about how that all works here:

https://alaveteli.org/docs/developers/i18n/

The core translation files (we also have translation projects for each theme and for the Pro-related translations) are stored in the locale directory of the Alaveteli codebase with a subdirectory for each language we support via Transifex; each folder is named for its corresponding locale tag.

Country specific language flavours (fr_BE, pt_BR etc) are a bit non-standard and not particularly well supported so we have some custom code to help us work with those. One of the locale-wrangling complexities we have to deal with is switching between the IETF (unicode)locale tag format adopted by Rails (and the HTML spec) that uses hyphens/dashes (en-US) and the POSIX (ISO 15897) locale format used by Transifex (sigh). See also: https://alaveteli.org/docs/developers/i18n/#technical-implementation-details

As far as possible, we've buried all that in the AlaveteliLocalization module that accepts either format as an input and provides the right one for the task in hand. That said, there are also some monkeypatches in lib/i18n_fixes.rb and we have some custom routing filter code in lib/routing_filters.rb (which uses the routing_filters gem to flexibly add a locale part to the route - most deployed sites are configured to not show the locale if using the default).

We've had problems in the past with setting a country-specific language as the default (again, the Belgian site's theme is a good test as it's set to have fr_BE as the default and nl_BE as the second language) so this should be tested as part of any major framework upgrade.

Known bugs

We're stuck on an old version of i18n because of the way they've made the loading of the translation files more efficient (it now only loads the files that are set in available_locales, avoiding traversing - and loading into memory - every subdirectory under the locales folder), which excludes the fallback translations that i18n-rails needs for framework message translations (e.g. names of days and months). So if we take the Belgian settings an example and set the default locale to fr_BE, i18n-rails may not have translations for Belgian French but can't fall back to general purpose French (fr) because it's not been loaded and raises an error instead. (If the default were available, the translations would revert to that instead which would provide an odd user experience - if you'd asked for Dutch but a phrase appeared in English or French instead, that would be odd but probably understandable, but the whole page being replaced with a generic error message is unusable.)

We're stuck on an old version of gettext (and therefore fast_gettext and gettext_i18n_rails) as version 3.0.0 introduced breaking changes so we'd need to revise our translation workflow and rewrite a bunch of customisations around translation generation and cleaning to be able to used the updated API. https://github.com/ruby-gettext/gettext/blob/master/doc/text/news.md#300-2013-08-31-version-3-0-0

There's a bug in setting and using current locale - we apply the default before considering the user's browser preference https://github.com/mysociety/alaveteli/issues/5080

Fallbacks don't work correctly when the default locale is an underscore locale https://github.com/mysociety/alaveteli/issues/5890