Implement ETag support #5298

greshilov · 2020-11-28T21:21:58Z

What do these changes do?

This PR adds etag property to the response object and if_match, if_none_match properties to the request object. Also it implements ETag support in static routes and fixes few bugs found along the way.

Despite the low priority of static files handling for aiohttp, I believe ETag and related headers has a wide application beyond this purpose.

Are there changes in behavior for the user?

Nope, no general changes in server behaviour.

Related issue number

Fixes #4594

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES folder
- name it <issue_id>.<type> for example (588.bugfix)
- if you don't have an issue_id change it to the pr id after creating the pr
- ensure type is one of the following:
  - .feature: Signifying a new feature.
  - .bugfix: Signifying a bug fix.
  - .doc: Signifying a documentation improvement.
  - .removal: Signifying a deprecation or removal of public API.
  - .misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

This change is

codecov · 2020-11-28T22:34:58Z

Codecov Report

Merging #5298 (699e959) into master (742a8b6) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5298      +/-   ##
==========================================
+ Coverage   97.17%   97.18%   +0.01%     
==========================================
  Files          41       41              
  Lines        8768     8849      +81     
  Branches     1404     1421      +17     
==========================================
+ Hits         8520     8600      +80     
- Misses        130      131       +1     
  Partials      118      118

Flag	Coverage Δ
unit	`97.07% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiohttp/__init__.py	`100.00% <100.00%> (ø)`
aiohttp/helpers.py	`96.77% <100.00%> (+0.09%)`	⬆️
aiohttp/web_fileresponse.py	`98.62% <100.00%> (-1.38%)`	⬇️
aiohttp/web_request.py	`95.92% <100.00%> (+0.19%)`	⬆️
aiohttp/web_response.py	`98.21% <100.00%> (+0.11%)`	⬆️
aiohttp/http_parser.py	`97.45% <0.00%> (+0.21%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 742a8b6...699e959. Read the comment docs.

aiohttp/helpers.py

serhiy-storchaka · 2020-12-03T21:53:01Z

aiohttp/web_request.py

+                value=ETAG_ANY,
+            )
+        else:
+            for match in LIST_QUOTED_ETAG_RE.finditer(etag_header):


It accepts also garbage like "1",spam"2".

Yeah, you're right.

To solve this problem I rejected the LIST_QUOTED_ETAG_RE approach, and switched to split + re.fullmatch.
Now only "1" would be matched from proposed garbage string. I've checked nginx behaviour and it's consistent with suggested solution. nginx matches etag values from header until first invalid is found. New approach is also 10x faster than old.

Main disadvantage is that memory consumption will probably increase though, but I don't think that really long If-Match, If-None-Match headers are so widespreaded.

aiohttp/web_request.py

aiohttp/helpers.py

aiohttp/web_request.py

serhiy-storchaka · 2020-12-03T22:05:36Z

aiohttp/web_request.py

+        if etag_header == ETAG_ANY:
+            yield ETag(
+                is_weak=False,
+                value=ETAG_ANY,


There is some inconsistency here. If ETag is "*", the value attribute will contain quotes. In all other cases they will be stripped.

Not quite. * is a kind of special value, which is used without quotes. So if someone somehow will use "*" instead of *, it will be considered as regular etag.

Unfortunatly ETag(value='*') produced from "*" is indistinguishable from one produced from *.
So, for example:

"*", "tag", "other-tag" -> (ETag(value='*'), ETag(value='other-tag')) * -> (ETag(value='*'), )

Maybe it's better to create separate type ETAG_ANY and change Optional[Tuple[ETag, ...]] to Optional[Union[ETAG_ANY, Tuple[ETag, ...]]]

"*", "tag", "other-tag" -> (ETag(value='*'), ETag(value='other-tag')) * -> ETAG_ANY

@asvetlov what do you think?

You are right.

aiohttp/web_response.py

asvetlov · 2020-12-05T08:14:30Z

@serhiy-storchaka thank you very much for the review!

@greshilov please address comments, they are very valuable.

greshilov · 2020-12-05T17:25:47Z

@serhiy-storchaka, @asvetlov thanks for the review! I appreciate your help.

I marked conversations as resolved where I used proposed solution without any changes, sorry if I shouldn't have done it by myself.

serhiy-storchaka · 2020-12-05T22:14:55Z

aiohttp/__init__.py

+from .helpers import (
+    BasicAuth as BasicAuth,
+    ChainMapProxy as ChainMapProxy,
+    ETag as ETag,
+)


Why not simply from .helpers import BasicAuth, ChainMapProxy, ETag?

I guess it's a kind of outer interface protection measures. Andrew can tell for sure.

No, we have no policy for such cases.
Usually, we use as only for importing under a different name if there is a conflict.

Please feel free to drop as.

@Dreamsorcerer

serhiy-storchaka · 2020-12-05T22:19:00Z

aiohttp/web_request.py

+        if etag_header == ETAG_ANY:
+            yield ETag(
+                is_weak=False,
+                value=ETAG_ANY,


You are right.

serhiy-storchaka · 2020-12-05T22:26:37Z

aiohttp/web_request.py

+                value=ETAG_ANY,
+            )
+        else:
+            for raw_etag in etag_header.split(","):


What if ETag contains comma? "1,2"

Actually the approach with LIST_QUOTED_ETAG_RE was not bad. You need just add |(.) at the end of the regexp. That group will be matched only if there is a garbage, so you can check it and raise an exception.

LIST_QUOTED_ETAG_RE = re.compile(fr"{_QUOTED_ETAG}(?:\s*,\s*|$)|(.)")

Yeah, completely forgot about this case while thinking about optimizations. Tests were updated to reflect this situation.

Thanks for the smart solution!

aiohttp/__init__.py

serhiy-storchaka · 2020-12-06T15:33:35Z

aiohttp/__init__.py

+from .helpers import (
+    BasicAuth,
+    ChainMapProxy,
+    ETag,
+)


Suggested change

from .helpers import (

BasicAuth,

ChainMapProxy,

ETag,

)

from .helpers import BasicAuth, ChainMapProxy, ETag

greshilov · 2020-12-09T23:38:24Z

I also have to mention, that according to rfc7232 If-Range header may contain ETag value. Current PR ignores that fact for now, because implementing ETag support for the If-Range header is not backward compatible, and should be separated to a different PR.

asvetlov · 2020-12-11T08:39:22Z

I also have to mention, that according to rfc7232 If-Range header may contain ETag value. Current PR ignores that fact for now, because implementing ETag support for the If-Range header is not backward compatible, and should be separated to a different PR.

Agree

asvetlov

LGTM.
Sorry for delay in the review

asvetlov · 2020-12-14T10:56:01Z

@greshilov please merge.

Please edit the commit message first: press "Squash and merge" -> edit the text to cleanup meaningless records about intermediate commits -> press "Confirm squash and merge".

greshilov · 2020-12-17T16:57:53Z

@greshilov please merge.

Please edit the commit message first: press "Squash and merge" -> edit the text to cleanup meaningless records about intermediate commits -> press "Confirm squash and merge".

Unfortunately, github doesn't show me this button. It says only users with write access can do this.

But I can squash commits manually before merge, if needed!

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

greshilov · 2021-01-31T19:52:29Z

@webknjaz, @Dreamsorcerer can you help me with squash/merging this? Andrew approved this PR but he has no spare time now.

I'm also planning to do a backport to 3.8 this week.

Dreamsorcerer · 2021-02-07T16:25:20Z

aiohttp/__init__.py

@@ -38,7 +38,7 @@
 )
 from .cookiejar import CookieJar as CookieJar, DummyCookieJar as DummyCookieJar
 from .formdata import FormData as FormData
-from .helpers import BasicAuth as BasicAuth, ChainMapProxy as ChainMapProxy
+from .helpers import BasicAuth, ChainMapProxy, ETag


Why are the imports changed here?

I'm assuming the style in this file follows the no implicit imports approach from Mypy:
https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-no-implicit-reexport

Although a little odd that it's not enforced in the Mypy tests if that is the case.

Those classes are already presented in the __all__ variable.
I agree about inconsistency with mypy style flags.

This change was previously discussed, I'll tag you there.

This is a great example of why style/formatting changes should be separated from the functional ones: a tiny disagreement can block a whole lot of legitimate changes, plus this makes it harder to notice important behavior changes.

I'll just leave this here: https://mtlynch.io/code-review-love/.

webknjaz · 2021-02-14T21:57:15Z

aiohttp/web_fileresponse.py

+        else:
+            return any(etag.value == etag_value for etag in etags if not etag.is_weak)


Nitpick, this could be less nested:

Suggested change

else:

return any(etag.value == etag_value for etag in etags if not etag.is_weak)

return any(etag.value == etag_value for etag in etags if not etag.is_weak)

webknjaz · 2021-02-14T21:58:45Z

aiohttp/web_fileresponse.py

+        self.etag = etag_value  # type: ignore
+        self.last_modified = last_modified  # type: ignore


It was agreed in other PRs to add specific error codes in brackets.

webknjaz · 2021-02-14T22:05:51Z

docs/spelling_wordlist.txt

@@ -29,6 +29,7 @@ Dict
 Discord
 Django
 Dup
+ETag


I'm almost sure that the spellchecker is case-insensitive.

webknjaz · 2021-02-14T22:06:55Z

tests/test_web_request.py

+
+
+@pytest.mark.parametrize(
+    "header,header_attr",


FTR it's usually better readable (though lesser known) to use iterables in parameterize:

Suggested change

"header,header_attr",

("header", "header_attr"),

webknjaz · 2021-02-14T22:10:07Z

tests/test_web_response.py

+)
+def test_etag_invalid_value_set(invalid_value) -> None:
+    resp = StreamResponse()
+    with pytest.raises(ValueError):


The best practice is to always include , match=r'...' argument demonstrating the intended exception reason more precisely.

webknjaz · 2021-02-14T22:12:48Z

@greshilov I didn't do a thorough review but commented with a few generic advices in several places. None of them are critical and since Andrew has already approved the patch, I'm going to merge it now. Feel free to send improvements in follow-up PRs.

webknjaz · 2021-02-14T22:26:00Z

@greshilov waiting for the backport to 3.8 now.

greshilov · 2021-02-14T23:06:52Z

@webknjaz sure, thank you!

@greshilov

This change adds an `etag` property to the response object and `if_match`, `if_none_match` properties to the request object. Also, it implements ETag support in static routes and fixes a few bugs found along the way. Refs: * https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 * https://tools.ietf.org/html/rfc7232#section-2.3 * https://tools.ietf.org/html/rfc7232#section-6 PR aio-libs#5298 by @greshilov Resolves aio-libs#4594 Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com> Co-Authored-By: Andrew Svetlov <andrew.svetlov@gmail.com>

@greshilov

This change adds an `etag` property to the response object and `if_match`, `if_none_match` properties to the request object. Also, it implements ETag support in static routes and fixes a few bugs found along the way. Refs: * https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 * https://tools.ietf.org/html/rfc7232#section-2.3 * https://tools.ietf.org/html/rfc7232#section-6 PR #5298 by @greshilov Resolves #4594 Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com> Co-Authored-By: Andrew Svetlov <andrew.svetlov@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

@greshilov

This change adds an `etag` property to the response object and `if_match`, `if_none_match` properties to the request object. Also, it implements ETag support in static routes and fixes a few bugs found along the way. Refs: * https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 * https://tools.ietf.org/html/rfc7232#section-2.3 * https://tools.ietf.org/html/rfc7232#section-6 PR aio-libs#5298 by @greshilov Resolves aio-libs#4594 Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

@greshilov

This change adds an `etag` property to the response object and `if_match`, `if_none_match` properties to the request object. Also, it implements ETag support in static routes and fixes a few bugs found along the way. Refs: * https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 * https://tools.ietf.org/html/rfc7232#section-2.3 * https://tools.ietf.org/html/rfc7232#section-6 PR aio-libs#5298 by @greshilov Resolves aio-libs#4594 Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

greshilov requested review from asvetlov and webknjaz as code owners November 28, 2020 21:21

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Nov 28, 2020

greshilov force-pushed the pr-4594 branch from 8ddc27f to 085bbc4 Compare November 28, 2020 22:30

asvetlov added the backport-3.8 label Nov 29, 2020

asvetlov reviewed Nov 29, 2020

View reviewed changes

aiohttp/helpers.py Outdated Show resolved Hide resolved

greshilov mentioned this pull request Dec 2, 2020

Naive datetime is interpreted as UTC when set last_modified #5304

Open

serhiy-storchaka reviewed Dec 3, 2020

View reviewed changes

greshilov force-pushed the pr-4594 branch 3 times, most recently from 0a01987 to b66aad1 Compare December 5, 2020 17:12

greshilov force-pushed the pr-4594 branch from b66aad1 to b39470d Compare December 5, 2020 17:32

serhiy-storchaka reviewed Dec 5, 2020

View reviewed changes

greshilov force-pushed the pr-4594 branch from b39470d to e6e979a Compare December 6, 2020 01:13

asvetlov reviewed Dec 6, 2020

View reviewed changes

aiohttp/__init__.py Outdated Show resolved Hide resolved

serhiy-storchaka reviewed Dec 6, 2020

View reviewed changes

greshilov force-pushed the pr-4594 branch 2 times, most recently from c4b70ef to 5f0ca5e Compare December 9, 2020 23:21

asvetlov approved these changes Dec 14, 2020

View reviewed changes

greshilov and others added 4 commits January 31, 2021 21:33

Implement ETag

ef014b1

Add if_match and if_none_match properties

76b35d5

FileResponse now supports ETag (aio-libs#4594)

e7ac04b

Update aiohttp/helpers.py

25a3cd6

asvetlov and others added 5 commits January 31, 2021 21:34

Update aiohttp/helpers.py

0fc7a59

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Update aiohttp/helpers.py

23aa15b

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Update ETag functionality

0762cfc

Update aiohttp/__init__.py

1ecac1c

Non-ASCII ETag tests and doc update

699e959

greshilov force-pushed the pr-4594 branch from 5f0ca5e to 699e959 Compare January 31, 2021 19:39

Dreamsorcerer reviewed Feb 7, 2021

View reviewed changes

webknjaz reviewed Feb 14, 2021

View reviewed changes

webknjaz merged commit ced553f into aio-libs:master Feb 14, 2021

greshilov added a commit to greshilov/aiohttp that referenced this pull request Mar 23, 2021

ETag fixes for aio-libs#5298

e331b56

greshilov added a commit to greshilov/aiohttp that referenced this pull request Mar 24, 2021

ETag fixes for aio-libs#5298

6756810

This was referenced Mar 24, 2021

ETag fixes PR #5298 #5561

Merged

[backport][3.8][PR#5298] ETag backport #5562

Merged

webknjaz pushed a commit that referenced this pull request Mar 24, 2021

ETag fixes for #5298 (#5561)

e74d40c

commonism pushed a commit to commonism/aiohttp that referenced this pull request Apr 27, 2021

ETag fixes for aio-libs#5298 (aio-libs#5561)

46c10de

commonism pushed a commit to commonism/aiohttp that referenced this pull request Apr 27, 2021

ETag fixes for aio-libs#5298 (aio-libs#5561)

7ddd204

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement ETag support #5298

Implement ETag support #5298

greshilov commented Nov 28, 2020 •

edited by asvetlov

codecov bot commented Nov 28, 2020 •

edited

serhiy-storchaka Dec 3, 2020

greshilov Dec 5, 2020 •

edited

serhiy-storchaka Dec 3, 2020

greshilov Dec 5, 2020

serhiy-storchaka Dec 5, 2020

asvetlov commented Dec 5, 2020

greshilov commented Dec 5, 2020 •

edited

serhiy-storchaka Dec 5, 2020

greshilov Dec 6, 2020

asvetlov Dec 6, 2020

greshilov Feb 14, 2021

serhiy-storchaka Dec 5, 2020

serhiy-storchaka Dec 5, 2020

greshilov Dec 6, 2020 •

edited

serhiy-storchaka Dec 6, 2020

greshilov Dec 6, 2020

greshilov commented Dec 9, 2020 •

edited

asvetlov commented Dec 11, 2020

asvetlov left a comment

asvetlov commented Dec 14, 2020

greshilov commented Dec 17, 2020 •

edited

greshilov commented Jan 31, 2021 •

edited

Dreamsorcerer Feb 7, 2021

greshilov Feb 14, 2021

webknjaz Feb 14, 2021

webknjaz Feb 14, 2021

webknjaz Feb 14, 2021

webknjaz Feb 14, 2021

webknjaz Feb 14, 2021 •

edited

webknjaz Feb 14, 2021

webknjaz commented Feb 14, 2021

webknjaz commented Feb 14, 2021

greshilov commented Feb 14, 2021

		else:
		return any(etag.value == etag_value for etag in etags if not etag.is_weak)

		self.etag = etag_value # type: ignore
		self.last_modified = last_modified # type: ignore

@@ @@ -29,6 +29,7 @@ Dict @@
               Discord
               Django
               Dup
+              ETag



		@pytest.mark.parametrize(
		"header,header_attr",

Implement ETag support #5298

Implement ETag support #5298

Conversation

greshilov commented Nov 28, 2020 • edited by asvetlov

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

codecov bot commented Nov 28, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

greshilov Dec 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asvetlov commented Dec 5, 2020

greshilov commented Dec 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greshilov Dec 6, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greshilov commented Dec 9, 2020 • edited

asvetlov commented Dec 11, 2020

asvetlov left a comment

Choose a reason for hiding this comment

asvetlov commented Dec 14, 2020

greshilov commented Dec 17, 2020 • edited

greshilov commented Jan 31, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz Feb 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Feb 14, 2021

webknjaz commented Feb 14, 2021

greshilov commented Feb 14, 2021

greshilov commented Nov 28, 2020 •

edited by asvetlov

codecov bot commented Nov 28, 2020 •

edited

greshilov Dec 5, 2020 •

edited

greshilov commented Dec 5, 2020 •

edited

greshilov Dec 6, 2020 •

edited

greshilov commented Dec 9, 2020 •

edited

greshilov commented Dec 17, 2020 •

edited

greshilov commented Jan 31, 2021 •

edited

webknjaz Feb 14, 2021 •

edited