checks: Add initial support for ICU MessageFormat (#6582)

* Add initial support for ICU MessageFormat.
* Implements a fairly comprehensive checking class.
* Adds documentation about the check to admin/checks.rst and user/checks.rst
* Includes a new dependency on `pyicumessageformat`.
* Add PrismJS syntax highlighting for ICU MessageFormat messages.

Co-authored-by: Michal Čihař <michal@cihar.com>

docs/admin/checks.rst

@@ -62,6 +62,10 @@ Here is a list of flags currently accepted:
     Define font-size for rendering checks, see :ref:`fonts`.
 ``font-spacing:SPACING``
     Define letter spacing for rendering checks, see :ref:`fonts`.
+``icu-flags:FLAGS``
+    Define flags for customizing the behavior of the :ref:`check-icu-message-format` quality check.
+``icu-tag-prefix:PREFIX``
+    Set a required prefix for XML tags for the :ref:`check-icu-message-format` quality check.
 ``placeholders:NAME:NAME2:...``
     Placeholder strings expected in translation, see :ref:`check-placeholders`.
 ``replacements:FROM:TO:FROM2:TO2...``
@@ -89,6 +93,8 @@ Here is a list of flags currently accepted:
     Enable the :ref:`check-es-format` quality check.
 ``i18next-interpolation``
     Enable the :ref:`check-i18next-interpolation` quality check.
+``icu-message-format``
+    Enable the :ref:`check-icu-message-format` quality check.
 ``java-format``
     Enable the :ref:`check-java-format` quality check.
 ``java-messageformat``
@@ -145,6 +151,8 @@ Here is a list of flags currently accepted:
     Skip the :ref:`check-es-format` quality check.
 ``ignore-i18next-interpolation``
     Skip the :ref:`check-i18next-interpolation` quality check.
+``ignore-icu-message-format``
+    Skip the :ref:`check-icu-message-format` quality check.
 ``ignore-java-format``
     Skip the :ref:`check-java-format` quality check.
 ``ignore-java-messageformat``

docs/user/checks.rst

@@ -242,8 +242,73 @@ i18next interpolation
    :ref:`check-formats`,
    `i18next interpolation <https://www.i18next.com/translation-function/interpolation>`_
 
-.. _check-java-format:
 
+.. _check-icu-message-format:
+
+ICU MessageFormat
+*****************
+
+.. versionadded:: 4.9
+
+:Summary: There are inconsistencies or syntax errors within ICU MessageFormat placeholders.
+:Scope: translated strings
+:Check class: ``weblate.checks.icu.ICUMessageFormatCheck``
+:Flag to enable: ``icu-message-format``
+:Flag to ignore: ``ignore-icu-message-format``
+:Interpolation example: ``There {number, plural, one {is one apple} other {are # apples}}.``
+
+This check has support for both pure ICU MessageFormat messages as well as ICU with simple
+XML tags. You can configure the behavior of this check by using ``icu-flags:*``, either by
+opting into XML support or by disabling certain sub-checks. For example, the following flag
+enables XML support while disabling validation of plural sub-messages:
+
+.. code-block::text
+
+  icu-message-format, icu-flags:xml:-plural_selectors
+
++---------------------------+------------------------------------------------------------+
+| ``xml``                   | Enable support for simple XML tags. By default, XML tags   |
+|                           | are parsed loosely. Stray ``<`` characters are ignored     |
+|                           | if they are not reasonably part of a tag.                  |
++---------------------------+------------------------------------------------------------+
+| ``strict-xml``            | Enable support for strict XML tags. All ``<`` characters   |
+|                           | must be escaped if they are not part of a tag.             |
++---------------------------+------------------------------------------------------------+
+| ``-highlight``            | Disable highlighting placeholders in the editor.           |
++---------------------------+------------------------------------------------------------+
+| ``-require_other``        | Disable requiring sub-messages to have an ``other``        |
+|                           | selector.                                                  |
++---------------------------+------------------------------------------------------------+
+| ``-submessage_selectors`` | Skip checking that sub-message selectors match the source. |
++---------------------------+------------------------------------------------------------+
+| ``-types``                | Skip checking that placeholder types match the source.     |
++---------------------------+------------------------------------------------------------+
+| ``-extra``                | Skip checking that no placeholders are present that were   |
+|                           | not present in the source string.                          |
++---------------------------+------------------------------------------------------------+
+| ``-missing``              | Skip checking that no placeholders are missing that were   |
+|                           | present in the source string.                              |
++---------------------------+------------------------------------------------------------+
+
+Additionally, when ``strict-xml`` is not enabled but ``xml`` is enabled, you can use the
+``icu-tag-prefix:PREFIX`` flag to require that all XML tags start with a specific string.
+For example, the following flag will only allow XML tags to be matched if they start with
+``<x:``:
+
+.. code-block::text
+
+  icu-message-format, icu-flags:xml, icu-tag-prefix:"x:"
+
+This would match ``<x:link>click here</x:link>`` but not ``<strong>this</strong>``.
+
+.. seealso::
+
+  :ref:`check-formats`,
+  `ICU: Formatting Messages <https://unicode-org.github.io/icu/userguide/format_parse/messages/>`_,
+  `Format.JS: Message Syntax <https://formatjs.io/docs/core-concepts/icu-syntax/>`_
+
+
+.. _check-java-format:
 
 Java format
 ***********

requirements.txt

@@ -24,6 +24,7 @@ Pillow>=6.0.0,<8.4.1
 pyahocorasick>=1.4,<1.5
 pycairo>=1.15.3
 pygobject>=3.27.0
+pyicumessageformat>=1.0.0,<1.1.0
 pyparsing>=2.4.0,<2.5.0
 python-dateutil>=2.8.1
 python-redis-lock>=3.6.0,<3.8.0

scripts/yarn-update

@@ -35,6 +35,7 @@ cp node_modules/prismjs/components/prism-core.js ../../weblate/static/vendor/pri
 cp node_modules/prismjs/components/prism-markup.js ../../weblate/static/vendor/prism/
 cp node_modules/prismjs/components/prism-rest.js ../../weblate/static/vendor/prism/
 cp node_modules/prismjs/components/prism-markdown.js ../../weblate/static/vendor/prism/
+cp node_modules/prismjs/components/prism-icu-message-format.js ../../weblate/static/vendor/prism/
 
 # Clipboard
 cp node_modules/clipboard/dist/clipboard.js ../../weblate/static/vendor/

weblate/checks/flags.py

@@ -65,6 +65,10 @@
 TYPED_FLAGS_ARGS["font-weight"] = single_value_flag(get_font_weight)
 TYPED_FLAGS["font-spacing"] = gettext_lazy("Font spacing")
 TYPED_FLAGS_ARGS["font-spacing"] = single_value_flag(int)
+TYPED_FLAGS["icu-flags"] = gettext_lazy("ICU MessageFormat Flags")
+TYPED_FLAGS_ARGS["icu-flags"] = multi_value_flag(str)
+TYPED_FLAGS["icu-tag-prefix"] = gettext_lazy("ICU MessageFormat Tag Prefix")
+TYPED_FLAGS_ARGS["icu-tag-prefix"] = single_value_flag(str)
 TYPED_FLAGS["priority"] = gettext_lazy("Priority")
 TYPED_FLAGS_ARGS["priority"] = single_value_flag(int)
 TYPED_FLAGS["max-length"] = gettext_lazy("Maximum length of translation")