Create timezone-aware datetimes when parsed as such #163

akaIDIOT · 2018-05-07T12:58:17Z

Includes commit by @cdavoren as his PR triggered this (changes effectively overwritten).

using datetime.timezone() for python3;
using minimal timezone() for python2.

Old behaviour:

>>> source = 'key: 2018-05-07T11:22:33+02:00'
>>> data = yaml.load(source)
>>> data['key']
datetime.datetime(2018, 5, 7, 9, 22, 33)

New behaviour:

>>> source = 'key: 2018-05-07T11:22:33+02:00'
>>> data = yaml.load(source)
>>> data['key']
datetime.datetime(2018, 5, 7, 11, 22, 33, tzinfo=datetime.timezone(datetime.timedelta(0, 7200)))

K0Te · 2018-05-17T10:58:13Z

This change would be very helpful !

akaIDIOT · 2018-05-29T08:56:44Z

To be honest, I'm not at all happy with the implementation of timezone for the py2 sources. Any great ideas on how to do that better (short of depending on pytz, guessing the pyyaml team won't like adding deps)?

seansfkelley · 2019-04-19T21:41:07Z

I just got bit by the lack of this behavior, so I'd love to see this PR get merged. What's wrong with the implementation as-is? If Python 2 doesn't have it, what else is feasible? I think you definitely don't want to add pytz.

Edit: @akaIDIOT you mentioned that you don't like the way this PR is written... what do you think of #251?

akaIDIOT · 2019-04-24T07:29:47Z

@seansfkelley they're basically the same thing I'd say. The timezone class is a little less messy than the one in this PR (and I personally hate dunder-private members :P), but I mainly care about the change in behaviour. The fact that multiple people are trying to fix this and others vote on the PR's tells me I'm not alone in this ;)

I'll clean up the timezone creation and rebase things on the current master, but it's up to the maintainers at this point.

akaIDIOT · 2019-04-24T09:07:36Z

Noticed the datetime regex matched a Z as timezone, also resulting in a naive, unshifted datetime. Made that use a zero-offset timezone for py2, datetime.timezone.utc for py3.

seansfkelley · 2019-04-24T18:53:22Z

@ingydotnet could you take a look at this PR or #251? I've had to rewrap PyYAML/switch away from it in some cases due to its datetime behavior. :(

jrd-nonlilly · 2019-08-05T17:46:02Z

It appears this problem has been around for the better part of a decade. Any hope of a merge?

mattseymour · 2019-09-19T11:59:45Z

Is there an ETA on supporting TZ date times. Currently 163, 113 and 251 are all open which try to resolve this issue. It seems to be an issue being faced by a lot of people and the fact 3 pull requests are open for such an issue highlights this.

It would be good if someone from the core team could shed some light on why these changes have taken a while to get reviewed so we may assist where possible; whether it is that the devs are busy or have reservations about the PR's. In either case, it would be good to start a conversation around this so we can all contribute to help push this change into master (sooner rather than later).

Cross-posting to other tickets.

perlpunk · 2019-12-09T17:25:59Z

Thanks! I think it would be nice to add a test for this.
I'm trying to get this into v5.3 or v5.4

OTOH, stil not sure if #251 would be better

akaIDIOT · 2019-12-10T10:09:55Z

I'm fine with merging either, or editing this, I care most about the fix :)

I'll rebase the PR and clean it up a bit, see if I can add a test. Leave the actual choice up to the devs.

On loading data, if timestamps have an ISO "+HH:MM" UTC offset then the resultant datetime is converted to UTC. This change adds that timezone information to the datetime objects. Importantly, this addresses a Django warning (and potential error) that appears when using both YAML fixtures in a timezone-aware project. It was raised as a Django issue (https://code.djangoproject.com/ticket/18867), but subsequently closed because the Django devs felt that this is a PyYAML problem.

Call datetime.datetime constructor once at return.

akaIDIOT · 2019-12-10T10:34:46Z

Test data already contains a bunch of datetimes with timezones that still seem to work. I'd be all for adding a test to make sure parsed values that have a timezone come out with a timezone attached, but the test code seems to want to just eval code and compare the values, not sure how easy it'd be to add a new assertion there somewhere...

perlpunk · 2019-12-15T16:54:06Z

I think I found a way to test for the timzones.
I will try and amend your PR...

edit: not sure that will work since your PR is made from master.
maybe I'll make my own PR after merging this

perlpunk · 2019-12-15T17:01:38Z

Thanks, merged!

After yaml#163, this adds some test data to check if the datetime objects return the correct timezone

perlpunk · 2019-12-15T17:14:49Z

@akaIDIOT I created #363 to add timezone tests. Maybe you could have a look if it makes sense?

After #163, this adds some test data to check if the datetime objects return the correct timezone

* On load, now use aware datetimes if possible On loading data, if timestamps have an ISO "+HH:MM" UTC offset then the resultant datetime is converted to UTC. This change adds that timezone information to the datetime objects. Importantly, this addresses a Django warning (and potential error) that appears when using both YAML fixtures in a timezone-aware project. It was raised as a Django issue (https://code.djangoproject.com/ticket/18867), but subsequently closed because the Django devs felt that this is a PyYAML problem. * Create timezone-aware datetime in timezone from data * Create timezone-aware datetime in timezone from data for python2 * Define better timezone implementation for python2 * Handle timezone "Z" for python 3 * Handle timezone "Z" for python 2 * Fix code structure for Python 3 Call datetime.datetime constructor once at return. * Fix code structure for Python 2 Call datetime.datetime constructor once at return.

After #163, this adds some test data to check if the datetime objects return the correct timezone

perlpunk · 2020-01-06T20:24:48Z

released https://pypi.org/project/PyYAML/5.3/

https://github.com/yaml/pyyaml/blob/d0d660d035905d9c49fc0f8dafb579d2cc68c0c8/CHANGES#L7 5.3.1 (2020-03-18) * yaml/pyyaml#386 -- Prevents arbitrary code execution during python/object/new constructor 5.3 (2020-01-06) * yaml/pyyaml#290 -- Use `is` instead of equality for comparing with `None` * yaml/pyyaml#270 -- fix typos and stylistic nit * yaml/pyyaml#309 -- Fix up small typo * yaml/pyyaml#161 -- Fix handling of __slots__ * yaml/pyyaml#358 -- Allow calling add_multi_constructor with None * yaml/pyyaml#285 -- Add use of safe_load() function in README * yaml/pyyaml#351 -- Fix reader for Unicode code points over 0xFFFF * yaml/pyyaml#360 -- Enable certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#359 -- Use full_load in yaml-highlight example * yaml/pyyaml#244 -- Document that PyYAML is implemented with Cython * yaml/pyyaml#329 -- Fix for Python 3.10 * yaml/pyyaml#310 -- increase size of index, line, and column fields * yaml/pyyaml#260 -- remove some unused imports * yaml/pyyaml#163 -- Create timezone-aware datetimes when parsed as such * yaml/pyyaml#363 -- Add tests for timezone 5.2 (2019-12-02) ------------------ * Repair incompatibilities introduced with 5.1. The default Loader was changed, but several methods like add_constructor still used the old default yaml/pyyaml#279 -- A more flexible fix for custom tag constructors yaml/pyyaml#287 -- Change default loader for yaml.add_constructor yaml/pyyaml#305 -- Change default loader for add_implicit_resolver, add_path_resolver * Make FullLoader safer by removing python/object/apply from the default FullLoader yaml/pyyaml#347 -- Move constructor for object/apply to UnsafeConstructor * Fix bug introduced in 5.1 where quoting went wrong on systems with sys.maxunicode <= 0xffff yaml/pyyaml#276 -- Fix logic for quoting special characters * Other PRs: yaml/pyyaml#280 -- Update CHANGES for 5.1

- Update from 3.13 to 6.0 - Update of rootfile - Changelog 6.0 (2021-10-13) * yaml/pyyaml#327 -- Change README format to Markdown * yaml/pyyaml#483 -- Add a test for YAML 1.1 types * yaml/pyyaml#497 -- fix float resolver to ignore `.` and `._` * yaml/pyyaml#550 -- drop Python 2.7 * yaml/pyyaml#553 -- Fix spelling of “hexadecimal” * yaml/pyyaml#556 -- fix representation of Enum subclasses * yaml/pyyaml#557 -- fix libyaml extension compiler warnings * yaml/pyyaml#560 -- fix ResourceWarning on leaked file descriptors * yaml/pyyaml#561 -- always require `Loader` arg to `yaml.load()` * yaml/pyyaml#564 -- remove remaining direct distutils usage 5.4.1 (2021-01-20) * yaml/pyyaml#480 -- Fix stub compat with older pyyaml versions that may unwittingly load it 5.4 (2021-01-19) * yaml/pyyaml#407 -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA * yaml/pyyaml#472 -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader * yaml/pyyaml#441 -- Fix memory leak in implicit resolver setup * yaml/pyyaml#392 -- Fix py2 copy support for timezone objects * yaml/pyyaml#378 -- Fix compatibility with Jython 5.3.1 (2020-03-18) * yaml/pyyaml#386 -- Prevents arbitrary code execution during python/object/new constructor 5.3 (2020-01-06) * yaml/pyyaml#290 -- Use `is` instead of equality for comparing with `None` * yaml/pyyaml#270 -- Fix typos and stylistic nit * yaml/pyyaml#309 -- Fix up small typo * yaml/pyyaml#161 -- Fix handling of __slots__ * yaml/pyyaml#358 -- Allow calling add_multi_constructor with None * yaml/pyyaml#285 -- Add use of safe_load() function in README * yaml/pyyaml#351 -- Fix reader for Unicode code points over 0xFFFF * yaml/pyyaml#360 -- Enable certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#359 -- Use full_load in yaml-highlight example * yaml/pyyaml#244 -- Document that PyYAML is implemented with Cython * yaml/pyyaml#329 -- Fix for Python 3.10 * yaml/pyyaml#310 -- Increase size of index, line, and column fields * yaml/pyyaml#260 -- Remove some unused imports * yaml/pyyaml#163 -- Create timezone-aware datetimes when parsed as such * yaml/pyyaml#363 -- Add tests for timezone 5.2 (2019-12-02) * Repair incompatibilities introduced with 5.1. The default Loader was changed, but several methods like add_constructor still used the old default yaml/pyyaml#279 -- A more flexible fix for custom tag constructors yaml/pyyaml#287 -- Change default loader for yaml.add_constructor yaml/pyyaml#305 -- Change default loader for add_implicit_resolver, add_path_resolver * Make FullLoader safer by removing python/object/apply from the default FullLoader yaml/pyyaml#347 -- Move constructor for object/apply to UnsafeConstructor * Fix bug introduced in 5.1 where quoting went wrong on systems with sys.maxunicode <= 0xffff yaml/pyyaml#276 -- Fix logic for quoting special characters * Other PRs: yaml/pyyaml#280 -- Update CHANGES for 5.1 5.1.2 (2019-07-30) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b2+ 5.1.1 (2019-06-05) * Re-release of 5.1 with regenerated Cython sources to build properly for Python 3.8b1 5.1 (2019-03-13) * yaml/pyyaml#35 -- Some modernization of the test running * yaml/pyyaml#42 -- Install tox in a virtualenv * yaml/pyyaml#45 -- Allow colon in a plain scalar in a flow context * yaml/pyyaml#48 -- Fix typos * yaml/pyyaml#55 -- Improve RepresenterError creation * yaml/pyyaml#59 -- Resolves #57, update readme issues link * yaml/pyyaml#60 -- Document and test Python 3.6 support * yaml/pyyaml#61 -- Use Travis CI built in pip cache support * yaml/pyyaml#62 -- Remove tox workaround for Travis CI * yaml/pyyaml#63 -- Adding support to Unicode characters over codepoint 0xffff * yaml/pyyaml#75 -- add 3.12 changelog * yaml/pyyaml#76 -- Fallback to Pure Python if Compilation fails * yaml/pyyaml#84 -- Drop unsupported Python 3.3 * yaml/pyyaml#102 -- Include license file in the generated wheel package * yaml/pyyaml#105 -- Removed Python 2.6 & 3.3 support * yaml/pyyaml#111 -- Remove commented out Psyco code * yaml/pyyaml#129 -- Remove call to `ord` in lib3 emitter code * yaml/pyyaml#149 -- Test on Python 3.7-dev * yaml/pyyaml#158 -- Support escaped slash in double quotes "\/" * yaml/pyyaml#175 -- Updated link to pypi in release announcement * yaml/pyyaml#181 -- Import Hashable from collections.abc * yaml/pyyaml#194 -- Reverting yaml/pyyaml#74 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and UnsafeLoader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not > 0xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> --git a/config/rootfiles/packages/python3-yaml b/config/rootfiles/packages/python3-yaml x 0870a2346..bd4009a08 100644 * yaml/pyyaml#195 -- Build libyaml on travis * yaml/pyyaml#196 -- Force cython when building sdist * yaml/pyyaml#254 -- Allow to turn off sorting keys in Dumper (2) * yaml/pyyaml#256 -- Make default_flow_style=False * yaml/pyyaml#257 -- Deprecate yaml.load and add FullLoader and Uns oader classes * yaml/pyyaml#261 -- Skip certain unicode tests when maxunicode not xffff * yaml/pyyaml#263 -- Windows Appveyor build Signed-off-by: Adolf Belka <adolf.belka@ipfire.org> Reviewed-by: Peter Müller <peter.mueller@ipfire.org>

penguinolog mentioned this pull request May 24, 2018

Any plans to release a new version? #154

Closed

seansfkelley mentioned this pull request Apr 20, 2019

Parse datetime timezone #251

Closed

This was referenced Sep 19, 2019

On load, create timezone-aware datetime objects if possible #113

Closed

Incompatible changes in version 5.1 #265

Open

perlpunk added this to Backlog in 5.3 Release Dec 9, 2019

cdavoren and others added 8 commits December 10, 2019 11:11

Create timezone-aware datetime in timezone from data

f7e81e6

Create timezone-aware datetime in timezone from data for python2

413a4df

Define better timezone implementation for python2

c78a22d

Handle timezone "Z" for python 3

09705f8

Handle timezone "Z" for python 2

31c176f

Fix code structure for Python 3

f2f4617

Call datetime.datetime constructor once at return.

Fix code structure for Python 2

5218216

Call datetime.datetime constructor once at return.

perlpunk approved these changes Dec 15, 2019

View reviewed changes

perlpunk changed the base branch from master to release/5.3 December 15, 2019 16:58

perlpunk merged commit 0cdef1d into yaml:release/5.3 Dec 15, 2019

perlpunk moved this from Backlog to Merged to release/5.3 in 5.3 Release Dec 15, 2019

perlpunk added a commit to perlpunk/pyyaml that referenced this pull request Dec 15, 2019

Add tests for timezone

5e15fd1

After yaml#163, this adds some test data to check if the datetime objects return the correct timezone

perlpunk mentioned this pull request Dec 15, 2019

Add tests for timezone #363

Merged

perlpunk added a commit that referenced this pull request Dec 16, 2019

Add tests for timezone

f532081

After #163, this adds some test data to check if the datetime objects return the correct timezone

perlpunk added a commit that referenced this pull request Dec 16, 2019

Add tests for timezone (#363)

74543d8

After #163, this adds some test data to check if the datetime objects return the correct timezone

perlpunk added a commit that referenced this pull request Dec 20, 2019

Add tests for timezone (#363)

4fcdcdb

After #163, this adds some test data to check if the datetime objects return the correct timezone

felixxm mentioned this pull request Jan 7, 2020

Fixed timezones tests for PyYAML 5.3+. django/django#12283

Merged

xiangce mentioned this pull request Jan 8, 2020

[Package dependency Error: PyYAML] TypeError: can't compare offset-naive and offset-aware datetimes RedHatInsights/insights-core#2397

Closed

This was referenced Jan 15, 2020

Fixes Build: Allow new datetime behavior in PyYAML 5.3 spec-first/connexion#1125

Merged

Build broken by datetime changes in PyYAML 5.3 spec-first/connexion#1126

Closed

polyzen mentioned this pull request Feb 29, 2020

Fix test_datetimes for PyYAML >=5.3 kislyuk/yq#89

Merged

asherf mentioned this pull request Apr 29, 2020

Upgrade PyYAML 5.1.2 to 5.3.1 pantsbuild/pants#9649

Merged

mercywediah mentioned this pull request Mar 26, 2021

Bump pyyaml from 3.13 to 5.4 mawy7/gra-asokwa-tincenter#5

Open

sync-by-unito bot mentioned this pull request Apr 12, 2021

Bump pyyaml from 5.1.2 to 5.4 lwinterface/panini_microservice_template#133

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create timezone-aware datetimes when parsed as such #163

Create timezone-aware datetimes when parsed as such #163

akaIDIOT commented May 7, 2018 •

edited

K0Te commented May 17, 2018

akaIDIOT commented May 29, 2018

seansfkelley commented Apr 19, 2019 •

edited

akaIDIOT commented Apr 24, 2019

akaIDIOT commented Apr 24, 2019

seansfkelley commented Apr 24, 2019

jrd-nonlilly commented Aug 5, 2019

mattseymour commented Sep 19, 2019

perlpunk commented Dec 9, 2019 •

edited

akaIDIOT commented Dec 10, 2019

akaIDIOT commented Dec 10, 2019

perlpunk commented Dec 15, 2019 •

edited

perlpunk commented Dec 15, 2019

perlpunk commented Dec 15, 2019

perlpunk commented Jan 6, 2020

Create timezone-aware datetimes when parsed as such #163

Create timezone-aware datetimes when parsed as such #163

Conversation

akaIDIOT commented May 7, 2018 • edited

K0Te commented May 17, 2018

akaIDIOT commented May 29, 2018

seansfkelley commented Apr 19, 2019 • edited

akaIDIOT commented Apr 24, 2019

akaIDIOT commented Apr 24, 2019

seansfkelley commented Apr 24, 2019

jrd-nonlilly commented Aug 5, 2019

mattseymour commented Sep 19, 2019

perlpunk commented Dec 9, 2019 • edited

akaIDIOT commented Dec 10, 2019

akaIDIOT commented Dec 10, 2019

perlpunk commented Dec 15, 2019 • edited

perlpunk commented Dec 15, 2019

perlpunk commented Dec 15, 2019

perlpunk commented Jan 6, 2020

akaIDIOT commented May 7, 2018 •

edited

seansfkelley commented Apr 19, 2019 •

edited

perlpunk commented Dec 9, 2019 •

edited

perlpunk commented Dec 15, 2019 •

edited