Syntax Error (unexpected colon) for list containing a URL

[KyleKing]

I initially reported this issue here: lyz-code/yamlfix#161

But the root cause appears to be that ruyaml doesn't handle the edge case that certain characters (colons and possibly others?) need to be escaped within a list. A reproducible example is below:

# Snippet based on: https://github.com/lyz-code/yamlfix/blob/master/src/yamlfix/services.py#L100-L117

from io import StringIO

import ruyaml

yaml_str = "- repos:\n  - command: [curl, 'https://github.com/pycontribs/ruyaml/issues/new']\n"
yaml = ruyaml.main.YAML()
yaml_dict = [*yaml.load_all(yaml_str)]
with StringIO() as string_stream:
    yaml.dump(yaml_dict, string_stream)
    yaml_dumped = string_stream.getvalue()

assert yaml_dumped ==  yaml_str
# But looks like this: '- repos:\n  - command: [curl, https://github.com/pycontribs/ruyaml/issues/new]\n'

I'm not very familiar with the YAML spec and the ruyaml codebase, but I could take a look at contributing if someone could point me in the right direction

[perlpunk]

this [curl, https://github.com/pycontribs/ruyaml/issues/new] is perfectly valid yaml

You can check several parsers with the yaml-runtimes containers, in several ways:

• use the gitpod environment
• run a docker command manually, e.g.
  echo '[curl, https://github.com/pycontribs/ruyaml/issues/new]' | docker run -i --rm yamlio/alpine-runtime-perl perl-refparser-event

Or you can use the playground https://play.yaml.io/main/parser?input=W2N1cmwsIGh0dHBzOi8vZ2l0aHViLmNvbS9weWNvbnRyaWJzL3J1eWFtbC9pc3N1ZXMvbmV3XQ==

A colon is only special if a whitespace is following (or if it is at the end of the line).

This was also fixed in pyyaml and libyaml a while ago:

• Allow colon in a plain scalar in a flow context yaml/pyyaml#45
• Allow colons in plain scalars inside flow collections yaml/libyaml#104

edit: btw, the YAML terminology for list is sequence, and the [...] syntax is called flow style

[KyleKing]

Thanks, I appreciate the extra background information!

Looks like my issue was with the pre-commit check-yaml hook when run with "safe"

https://github.com/pre-commit/pre-commit-hooks/blob/40cc31b02ceb72045509576e51d9c82503751422/pre_commit_hooks/check_yaml.py

>>> import ruamel.yaml
>>>
>>> yaml_str = "- repos:\n  - command: [curl, https://github.com/pycontribs/ruyaml/issues/new]\n"
>>>
>>> yaml = ruamel.yaml.YAML(typ='safe')
>>> yaml_dict = yaml.load(yaml_str)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kyleking/Developer/wip/calcipy/.venv/lib/python3.10/site-packages/ruamel/yaml/main.py", line 434, in load
    return constructor.get_single_data()
  File "/Users/kyleking/Developer/wip/calcipy/.venv/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 706, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 724, in _ruamel_yaml.CParser._compose_document
  File "_ruamel_yaml.pyx", line 773, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 850, in _ruamel_yaml.CParser._compose_sequence_node
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 773, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 850, in _ruamel_yaml.CParser._compose_sequence_node
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 773, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 852, in _ruamel_yaml.CParser._compose_sequence_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.scanner.ScannerError: while scanning a plain scalar
  in "<unicode string>", line 2, column 21
found unexpected ':'
  in "<unicode string>", line 2, column 26
>>>
>>> yaml = ruamel.yaml.YAML(typ='safe')
>>> yaml_dicts = [*yaml.parse(yaml_str)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kyleking/Developer/wip/calcipy/.venv/lib/python3.10/site-packages/ruamel/yaml/main.py", line 348, in parse
    while parser.check_event():
  File "_ruamel_yaml.pyx", line 675, in _ruamel_yaml.CParser.check_event
  File "_ruamel_yaml.pyx", line 529, in _ruamel_yaml.CParser._parse
ruamel.yaml.scanner.ScannerError: while scanning a plain scalar
  in "<unicode string>", line 2, column 21
found unexpected ':'
  in "<unicode string>", line 2, column 26
>>>
>>>
>>> # No error when not set to "safe"
>>> yaml = ruamel.yaml.YAML()
>>> yaml_dict = yaml.load(yaml_str)
>>> print(yaml_dict)
[ordereddict([('repos', [ordereddict([('command', ['curl', 'https://github.com/pycontribs/ruyaml/issues/new'])])])])]
>>>

[tales-aparecida]

Just so future readers are aware, check-yaml hook currently uses ruamel instead of ruyaml, and there's a ticket related to this issue at https://sourceforge.net/p/ruamel-yaml/tickets/248/, where they give a workaround of using the slower Python interpreter, for an issue apparently related to libyaml.
The workaround to fix the hook itself can be to downgrade the hook to v2.1.0, when it was still using pyYAML, using args: [--unsafe] depending on your use case.

 -   repo: https://github.com/pre-commit/pre-commit-hooks                        
     rev: v2.1.0                                                                 
     hooks:                                                                      
     -   id: check-yaml                                                          
         args: [--unsafe]