Allow colon in a plain scalar in a flow context #45

dbeer1 · 2016-12-07T19:54:48Z

This pull is analogous to yaml/libyaml#28. It changes the behavior of pyyaml the same way the other pull changes the behavior of libyaml.

sigmavirus24 · 2016-12-07T22:13:54Z

Hey @dbeer1!

Refresh my memory, is this valid for scalars even in 1.1 or just 1.2? At the moment, pyyaml and libyaml only support 1.1.

sigmavirus24 · 2016-12-07T22:14:47Z

tests/data/spec-08-13.canonical

@@ -1,7 +1,7 @@
 %YAML 1.1
 ---
 !!map {
-  ? !!str "foo"
+  ? !!str "foo :"


Did this fail prior to this change?

It passed, but the key was parsed as "foo". Note that pyyaml and libyaml behave differently in this edge case, libyaml rejects any colon in a flow context without a space afterwards, while pyyaml allows ',[]{}' after a colon as well.

Compare https://github.com/yaml/pyyaml/blob/master/lib/yaml/scanner.py#L1301 vs https://github.com/yaml/libyaml/blob/master/src/scanner.c#L3438

I reverted this test and behavior with regard to empty values in flow mappings in the latest commit, fyi.

adrienverge · 2016-12-09T10:27:04Z

I just check in the specs, and it appears to be the same for 1.1 and 1.2. From 1.1:

Plain scalars can never contain any tab characters. They also must not contain the “: ” and “ #” character sequences as these combinations cause ambiguity with key: value pairs and comments. Inside flow collections, plain scalars are further restricted to avoid containing the “[”, “]”, “{”, “}” and “,” characters as these would cause ambiguity with the flow collection structure (hence the need for the flow-in context and the flow-out context).

So ": " (notice the space) is forbidden, but ":" isn't.

Related issue: adrienverge/yamllint#30

adrienverge · 2017-02-06T13:34:06Z

Hi @sigmavirus24, do you have any update on this?

ingydotnet · 2017-02-08T19:42:08Z

Kirill strayed from the spec a few times when he thought it was really not the right thing to do. I suspect this might be the case. {foo:bar} is pretty ambiguous. Even {url:http://foo.io} is weird looking at best. However [ http://127.0.0.1:1234, http://127.0.0.1:2345 ] is not.

I just tested this using the new YAML Editor: http://pasteboard.co/vYxs4oX8D.png

Seems like most frameworks accept it, except the libyaml ones. Since it is in the spec and EveryonesDoingIt™, I move to add this to libyaml and all its bindings.

sigmavirus24 · 2017-02-08T19:51:07Z

Thanks for confirming that detail @ingydotnet

dbeer1 · 2017-02-08T20:21:50Z

Thanks! Note that the C binding pull request (yaml/libyaml#28) needs to be updated to follow the modified logic here with regard to a missing value after a colon, eg {a:} -> { "a" : null } instead of {a:} -> { "a:" : null }

bobvandevijver · 2017-02-27T17:47:50Z

@sigmavirus24 Can a new release be drafted with this change? I'm waiting for it so my I can update my docker containers without moving from the simple pip install command 😃

tobias-93 · 2017-07-26T20:54:17Z

@sigmavirus24 Would it be possible to make a new release with these changes? Thank you!

asomov · 2017-12-23T15:10:06Z

Hi all, I see that the change was already merged.
I think that it is still incomplete. It should be finished before releasing.

No new tests have been added. There should be at least a test for {a:1} and {a:} (because they are counter intuitive)
The file spec-08-13.skip-ext should be removed to let the test fail. Then either the implementation or spec-08-13.canonical should be adapted.

We have a similar request in SnakeYAML
I would prefer to solve the questions before SnakeYAML follows the change.

This reverts commit c5b135f.

asomov · 2018-03-03T11:05:16Z

The same change is implemented in SnakeYAML: https://bitbucket.org/asomov/snakeyaml/issues/397

ingydotnet · 2018-06-24T22:29:49Z

@asomov The PyYAML release of this is being planned to happen in the next 48 hours. Would you mind dropping by #pyyaml on irc.freenode.net to discuss?

koitsu · 2020-03-18T00:26:47Z

Commit c5b135f causes a syntax error on Python 3.2.3. Yes I know, 3.2.3 is over 9 years old, but I wanted to at least note this "regression" of sort somewhere in case others come across it. I have not tested Python 2.x.

$ python3
Python 3.2.3 (default, Nov 17 2016, 01:04:00)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/yaml/__init__.py", line 8, in <module>
    from .loader import *
  File "/usr/lib/python3/yaml/loader.py", line 5, in <module>
    from .scanner import *
  File "/usr/lib/python3/yaml/scanner.py", line 1294
    + (u',[]{}' if self.flow_level else u''))\
              ^
SyntaxError: invalid syntax

- Update from 3.13 to 6.0 - Update of rootfile - Changelog 6.0 (2021-10-13) * yaml/pyyaml#327 -- Change README format to Markdown * yaml/pyyaml#483 -- Add a test for YAML 1.1 types * yaml/pyyaml#497 -- fix float resolver to ignore `.` and `._` * yaml/pyyaml#550 -- drop Python 2.7 * yaml/pyyaml#553 -- Fix spelling of “hexadecimal” * yaml/pyyaml#556 -- fix representation of Enum subclasses * yaml/pyyaml#557 -- fix libyaml extension compiler warnings * yaml/pyyaml#560 -- fix ResourceWarning on leaked file descriptors * yaml/pyyaml#561 -- always require `Loader` arg to `yaml.load()` * yaml/pyyaml#564 -- remove remaining direct distutils usage 5.4.1 (2021-01-20) * yaml/pyyaml#480 -- Fix stub compat with older pyyaml versions that may unwittingly load it 5.4 (2021-01-19) * yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA * yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader * yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup * yaml/pyyaml#392 -- Fix py2 copy support for timezone objects * yaml/pyyaml#378 -- Fix compatibility with Jython 5.3.1 (2020-03-18) * yaml/pyyaml#386 -- Prevents arbitrary code execution during python/object/new constructor 5.3 (2020-01-06) * yaml/pyyaml#290 -- Use `is` instead of equality for comparing with `None` * yaml/pyyaml#270 -- Fix typos and stylistic nit * yaml/pyyaml#309 -- Fix up small typo * yaml/pyyaml#161 -- Fix handling of __slots__ * yaml/pyyaml#358 -- Allow calling add_multi_constructor with None * yaml/pyyaml#285 -- Add use of safe_load() function in README * yaml/pyyaml#351 -- Fix reader for Unicode code points over 0xFFFF * yaml/pyyaml#360 -- Enable certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#359 -- Use full_load in yaml-highlight example * yaml/pyyaml#244 -- Document that PyYAML is implemented with Cython * yaml/pyyaml#329 -- Fix for Python 3.10 * yaml/pyyaml#310 -- Increase size of index, line, and column fields * yaml/pyyaml#260 -- Remove some unused imports * yaml/pyyaml#163 -- Create timezone-aware datetimes when parsed as such * yaml/pyyaml#363 -- Add tests for timezone 5.2 (2019-12-02) * Repair incompatibilities introduced with 5.1. The default Loader was changed, but several methods like add_constructor still used the old default yaml/pyyaml#279 -- A more flexible fix for custom tag constructors yaml/pyyaml#287 -- Change default loader for yaml.add_constructor yaml/pyyaml#305 -- Change default loader for add_implicit_resolver, add_path_resolver * Make FullLoader safer by removing python/object/apply from the default FullLoader yaml/pyyaml#347 -- Move constructor for object/apply to UnsafeConstructor * Fix bug introduced in 5.1 where quoting went wrong on systems with sys.maxunicode <= 0xffff yaml/pyyaml#276 -- Fix logic for quoting special characters * Other PRs: yaml/pyyaml#280 -- Update CHANGES for 5.1 5.1.2 (2019-07-30) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b2+ 5.1.1 (2019-06-05) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b1 5.1 (2019-03-13) * yaml/pyyaml#35 -- Some modernization of the test running * yaml/pyyaml#42 -- Install tox in a virtualenv * yaml/pyyaml#45 -- Allow colon in a plain scalar in a flow context * yaml/pyyaml#48 -- Fix typos * yaml/pyyaml#55 -- Improve RepresenterError creation * yaml/pyyaml#59 -- Resolves #57, update readme issues link * yaml/pyyaml#60 -- Document and test Python 3.6 support * yaml/pyyaml#61 -- Use Travis CI built in pip cache support * yaml/pyyaml#62 -- Remove tox workaround for Travis CI * yaml/pyyaml#63 -- Adding support to Unicode characters over codepoint 0xffff * yaml/pyyaml#75 -- add 3.12 changelog * yaml/pyyaml#76 -- Fallback to Pure Python if Compilation fails * yaml/pyyaml#84 -- Drop unsupported Python 3.3 * yaml/pyyaml#102 -- Include license file in the generated wheel package * yaml/pyyaml#105 -- Removed Python 2.6 & 3.3 support * yaml/pyyaml#111 -- Remove commented out Psyco code * yaml/pyyaml#129 -- Remove call to `ord` in lib3 emitter code * yaml/pyyaml#149 -- Test on Python 3.7-dev * yaml/pyyaml#158 -- Support escaped slash in double quotes "\/" * yaml/pyyaml#175 -- Updated link to pypi in release announcement * yaml/pyyaml#181 -- Import Hashable from collections.abc * yaml/pyyaml#194 -- Reverting yaml/pyyaml#74 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and UnsafeLoader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> --git a/config/rootfiles/packages/python3-yaml b/config/rootfiles/packages/python3-yaml x 0870a2346..bd4009a08 100644 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and Uns oader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> Reviewed-by: Peter Müller <peter.mueller@ipfire.org>

@tamere-allo-peter

Commit c268a82 "key-duplicates: Don't crash on redundant closing brackets or braces" fixed a problem but introduced another one: it crashes on systems with (I guess) an old version of PyYAML. This is probably linked to the "Allow colon in a plain scalar in a flow context" issue on PyYAML [1]. For example, this problem happens on CentOS 8: FAIL: test_disabled (tests.rules.test_key_duplicates.KeyDuplicatesTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "…/tests/rules/test_key_duplicates.py", line 90, in test_disabled '{a:1, b:2}}\n', conf, problem=(2, 11, 'syntax')) File "…/tests/common.py", line 54, in check self.assertEqual(real_problems, expected_problems) AssertionError: Lists differ: … - [2:3: syntax error: found unexpected ':' (syntax)] + [2:11: <no description>] I propose to simply fix the *space following a colon* problem, since it's not related to what the original author @tamere-allo-peter tried to fix. [1]: yaml/pyyaml#45

@tamere-allo-peter

Commit c268a82 "key-duplicates: Don't crash on redundant closing brackets or braces" fixed a problem but introduced another one: it crashes on systems with (I guess) an old version of PyYAML. This is probably linked to the "Allow colon in a plain scalar in a flow context" issue on PyYAML [1]. For example, this problem happens on CentOS 8: FAIL: test_disabled (tests.rules.test_key_duplicates.KeyDuplicatesTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "…/tests/rules/test_key_duplicates.py", line 90, in test_disabled '{a:1, b:2}}\n', conf, problem=(2, 11, 'syntax')) File "…/tests/common.py", line 54, in check self.assertEqual(real_problems, expected_problems) AssertionError: Lists differ: … - [2:3: syntax error: found unexpected ':' (syntax)] + [2:11: <no description>] I propose to simply fix the *space following a colon* problem, since it's not related to what the original author @tamere-allo-peter tried to fix. [1]: yaml/pyyaml#45

Quote scalars with colons in flow style collection SUMMARY Older versions of libyaml and pyyaml don't support colons in scalars that appear in flow style collections. This adds quotes to the handful of cases where they are used in the integration tests. This was uncovered by downstream testing. I am only able to reproduce the problem on the supported ee-minimal-rhel8 image. I believe this is because even though pyyaml should be recent enough, it has likely been compiled against an older version of libyaml that does not have the fix. See: yaml/libyaml#104 yaml/pyyaml#45 It's worth noting that running ansible-lint --fix will undo these changes, which is how I think they were made in the first place. ISSUE TYPE Bugfix Pull Request COMPONENT NAME ADDITIONAL INFORMATION Reviewed-by: Bikouo Aubin Reviewed-by: GomathiselviS

Quote scalars with colons in flow style collection SUMMARY Older versions of libyaml and pyyaml don't support colons in scalars that appear in flow style collections. This adds quotes to the handful of cases where they are used in the integration tests. This was uncovered by downstream testing. I am only able to reproduce the problem on the supported ee-minimal-rhel8 image. I believe this is because even though pyyaml should be recent enough, it has likely been compiled against an older version of libyaml that does not have the fix. See: yaml/libyaml#104 yaml/pyyaml#45 It's worth noting that running ansible-lint --fix will undo these changes, which is how I think they were made in the first place. ISSUE TYPE Bugfix Pull Request COMPONENT NAME ADDITIONAL INFORMATION Reviewed-by: Bikouo Aubin Reviewed-by: GomathiselviS (cherry picked from commit 68fb0a3)

[PR #2014/68fb0a31 backport][stable-7] Quote scalars with colons in flow style collection This is a backport of PR #2014 as merged into main (68fb0a3). SUMMARY Older versions of libyaml and pyyaml don't support colons in scalars that appear in flow style collections. This adds quotes to the handful of cases where they are used in the integration tests. This was uncovered by downstream testing. I am only able to reproduce the problem on the supported ee-minimal-rhel8 image. I believe this is because even though pyyaml should be recent enough, it has likely been compiled against an older version of libyaml that does not have the fix. See: yaml/libyaml#104 yaml/pyyaml#45 It's worth noting that running ansible-lint --fix will undo these changes, which is how I think they were made in the first place. ISSUE TYPE Bugfix Pull Request COMPONENT NAME ADDITIONAL INFORMATION Reviewed-by: Mark Chappell

Allow colon in a plain scalar in a flow context

91f38ed

sigmavirus24 self-assigned this Dec 7, 2016

sigmavirus24 reviewed Dec 7, 2016

View reviewed changes

Restore behavior of flow mapping with empty value

53e22ad

dbeer1 mentioned this pull request Dec 8, 2016

colon can be in a plain scalar yaml/libyaml#28

Closed

adrienverge mentioned this pull request Dec 9, 2016

Symfony routing-file validation adrienverge/yamllint#30

Closed

sigmavirus24 merged commit c5b135f into yaml:master Feb 8, 2017

dbeer1 deleted the feature/colon_in_flow_context branch July 19, 2017 17:12

asomov added a commit to asomov/pyyaml that referenced this pull request Mar 2, 2018

Revert "Allow colon in a plain scalar in a flow context (yaml#45)"

d6e157b

This reverts commit c5b135f.

asomov mentioned this pull request Mar 2, 2018

Revert "Allow colon in a plain scalar in a flow context (#45)" #140

Closed

ingydotnet added this to Added in 5.1 Release Nov 8, 2018

ingydotnet removed this from Added to release/5/1 branch in 5.1 Release Feb 23, 2019

perlpunk mentioned this pull request Mar 14, 2019

Incompatible changes in version 5.1 #265

Open

dandavison mentioned this pull request Oct 21, 2019

Bump pyyaml from 3.13 to 5.1 dandavison/facet#5

Merged

RhythmicGaming mentioned this pull request Jan 15, 2020

Bump pyyaml from 3.13 to 5.1 in /data_builders/edhrec DEGoodmanWilson/magique#9

Closed

Ciemaar mentioned this pull request May 30, 2020

Bump pyyaml from 3.12 to 5.1 Ciemaar/flock#5

Merged

This was referenced Mar 9, 2021

build(deps-dev): bump pyyaml from 3.12 to 5.1 in /requirements bilalzee/serverless-application-model#2

Closed

Bump pyyaml from 3.13 to 5.1 lifeready/dvpwa#3

Closed

kddejong mentioned this pull request Apr 19, 2021

Issue parsing colons on Linux aws-cloudformation/cfn-lint#1984

Closed

perlpunk mentioned this pull request Feb 21, 2022

Syntax Error (unexpected colon) for list containing a URL pycontribs/ruyaml#86

Closed

adrienverge mentioned this pull request Jul 8, 2022

key-duplicates: Fix failing test for missing space after colon adrienverge/yamllint#479

Merged

gravesm mentioned this pull request Mar 6, 2024

Quote scalars with colons in flow style collection ansible-collections/amazon.aws#2014

Merged

patchback bot mentioned this pull request Mar 13, 2024

[PR #2014/68fb0a31 backport][stable-7] Quote scalars with colons in flow style collection ansible-collections/amazon.aws#2017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow colon in a plain scalar in a flow context #45

Allow colon in a plain scalar in a flow context #45

dbeer1 commented Dec 7, 2016

sigmavirus24 commented Dec 7, 2016

sigmavirus24 Dec 7, 2016

dbeer1 Dec 8, 2016 •

edited

dbeer1 Dec 8, 2016

adrienverge commented Dec 9, 2016 •

edited

adrienverge commented Feb 6, 2017

ingydotnet commented Feb 8, 2017

sigmavirus24 commented Feb 8, 2017

dbeer1 commented Feb 8, 2017

bobvandevijver commented Feb 27, 2017

tobias-93 commented Jul 26, 2017

asomov commented Dec 23, 2017

asomov commented Mar 3, 2018

ingydotnet commented Jun 24, 2018

koitsu commented Mar 18, 2020

Allow colon in a plain scalar in a flow context #45

Allow colon in a plain scalar in a flow context #45

Conversation

dbeer1 commented Dec 7, 2016

sigmavirus24 commented Dec 7, 2016

sigmavirus24 Dec 7, 2016

Choose a reason for hiding this comment

dbeer1 Dec 8, 2016 • edited

Choose a reason for hiding this comment

dbeer1 Dec 8, 2016

Choose a reason for hiding this comment

adrienverge commented Dec 9, 2016 • edited

adrienverge commented Feb 6, 2017

ingydotnet commented Feb 8, 2017

sigmavirus24 commented Feb 8, 2017

dbeer1 commented Feb 8, 2017

bobvandevijver commented Feb 27, 2017

tobias-93 commented Jul 26, 2017

asomov commented Dec 23, 2017

asomov commented Mar 3, 2018

ingydotnet commented Jun 24, 2018

koitsu commented Mar 18, 2020

dbeer1 Dec 8, 2016 •

edited

adrienverge commented Dec 9, 2016 •

edited