Tests failing on tests/test_stream_json.py

[iagocq]

Checklist

• This is a bug report and not a different kind of issue
• I have read the contribution guidelines
• I have checked the list of open and recently closed bug reports
• I have checked the commit log of the master branch

Streamlink version

Latest stable release

Description

The test_hls_stream and test_http_stream tests on tests/test_stream_json.py fail if there's an extra encoding added to the Accept-Encoding header (brotli in this case).

I'm currently trying to find out why that is being added to the header in the first place in my system (NixOS), but it would be nice if the test was more lenient with this kind of difference.

Debug log

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /build/streamlink-2.4.0
plugins: requests-mock-1.9.3
collected 2444 items

tests/test_api_http_session.py ....                                      [  0%]
tests/test_api_validate.py ..................................            [  1%]
tests/test_buffer.py ............                                        [  2%]
tests/test_cache.py ...............                                      [  2%]
tests/test_cli_main.py ...............................sss                [  4%]
tests/test_cli_playerout.py ..ss.s                                       [  4%]
tests/test_cli_util_progress.py ..                                       [  4%]
tests/test_cli_utils_formatter.py ....                                   [  4%]
tests/test_cli_utils_path.py .................................s.s.s.     [  6%]
tests/test_cmdline.py .....ssssss                                        [  6%]
tests/test_cmdline_player_fifo.py .sss                                   [  6%]
tests/test_cmdline_title.py .....sssssss                                 [  7%]
tests/test_console.py ...............                                    [  7%]
tests/test_lazy_formatter.py ........                                    [  8%]
tests/test_localization.py ................sssssssssssssssss             [  9%]
tests/test_log.py .........                                              [  9%]
tests/test_meta.py ..                                                    [  9%]
tests/test_options.py ...............                                    [ 10%]
tests/test_output.py .......................                             [ 11%]
tests/test_plugin.py ..........                                          [ 11%]
tests/test_plugin_utils.py ......                                        [ 12%]
tests/test_plugins.py .................................................. [ 14%]
........................................................................ [ 17%]
......................................                                   [ 18%]
tests/test_plugins_input.py .....                                        [ 18%]
tests/test_plugins_meta.py ......                                        [ 19%]
tests/test_session.py .....................                              [ 20%]
tests/test_stream_file.py ..                                             [ 20%]
tests/test_stream_json.py ...FF.                                         [ 20%]
tests/test_streamlink_api.py .....                                       [ 20%]
tests/test_utils.py ..................                                   [ 21%]
tests/test_utils_args.py ...........                                     [ 21%]
tests/test_utils_crypto.py ..                                            [ 21%]
tests/test_utils_named_pipe.py ......ssss                                [ 22%]
tests/test_utils_times.py ..                                             [ 22%]
tests/test_utils_url.py ....                                             [ 22%]
tests/plugins/test_abematv.py ................                           [ 23%]
tests/plugins/test_abweb.py .........                                    [ 23%]
tests/plugins/test_adultswim.py ..........                               [ 23%]
tests/plugins/test_afreeca.py .............                              [ 24%]
tests/plugins/test_albavision.py ............                            [ 24%]
tests/plugins/test_app17.py .....                                        [ 25%]
tests/plugins/test_ard_live.py .......                                   [ 25%]
tests/plugins/test_ard_mediathek.py ........                             [ 25%]
tests/plugins/test_artetv.py .....................                       [ 26%]
tests/plugins/test_atresplayer.py ........                               [ 26%]
tests/plugins/test_bbciplayer.py ........                                [ 27%]
tests/plugins/test_bfmtv.py .................                            [ 27%]
tests/plugins/test_bigo.py ....................                          [ 28%]
tests/plugins/test_bilibili.py .....                                     [ 29%]
tests/plugins/test_bloomberg.py ..............                           [ 29%]
tests/plugins/test_booyah.py ........................                    [ 30%]
tests/plugins/test_brightcove.py ......                                  [ 30%]
tests/plugins/test_btv.py .......                                        [ 31%]
tests/plugins/test_cbsnews.py .........                                  [ 31%]
tests/plugins/test_cdnbg.py ..............................               [ 32%]
tests/plugins/test_ceskatelevize.py ..........                           [ 33%]
tests/plugins/test_cinergroup.py ..........                              [ 33%]
tests/plugins/test_clubbingtv.py ........                                [ 33%]
tests/plugins/test_cnews.py ........                                     [ 34%]
tests/plugins/test_crunchyroll.py ..............                         [ 34%]
tests/plugins/test_dailymotion.py ........                               [ 35%]
tests/plugins/test_dash.py ..............                                [ 35%]
tests/plugins/test_delfi.py .........                                    [ 36%]
tests/plugins/test_deutschewelle.py ..........                           [ 36%]
tests/plugins/test_dlive.py .......                                      [ 36%]
tests/plugins/test_dogan.py ...............................              [ 37%]
tests/plugins/test_dogus.py .........                                    [ 38%]
tests/plugins/test_drdk.py ..........                                    [ 38%]
tests/plugins/test_earthcam.py ......                                    [ 38%]
tests/plugins/test_egame.py .........                                    [ 39%]
tests/plugins/test_eltrecetv.py ........                                 [ 39%]
tests/plugins/test_euronews.py ...................                       [ 40%]
tests/plugins/test_facebook.py ..........                                [ 40%]
tests/plugins/test_filmon.py ......................                      [ 41%]
tests/plugins/test_foxtr.py .....                                        [ 41%]
tests/plugins/test_funimationnow.py .........                            [ 42%]
tests/plugins/test_galatasaraytv.py ........                             [ 42%]
tests/plugins/test_garena.py ........                                    [ 43%]
tests/plugins/test_goltelevision.py ........                             [ 43%]
tests/plugins/test_goodgame.py ........                                  [ 43%]
tests/plugins/test_googledrive.py .....                                  [ 43%]
tests/plugins/test_gulli.py ............                                 [ 44%]
tests/plugins/test_huajiao.py .....                                      [ 44%]
tests/plugins/test_huya.py ........                                      [ 44%]
tests/plugins/test_idf1.py ..........                                    [ 45%]
tests/plugins/test_invintus.py .........                                 [ 45%]
tests/plugins/test_kugou.py ..........                                   [ 46%]
tests/plugins/test_latina.py ......                                      [ 46%]
tests/plugins/test_linelive.py .......                                   [ 46%]
tests/plugins/test_live_russia_tv.py ........                            [ 46%]
tests/plugins/test_liveme.py .........                                   [ 47%]
tests/plugins/test_livestream.py .......                                 [ 47%]
tests/plugins/test_lrt.py ............                                   [ 48%]
tests/plugins/test_ltv_lsm_lv.py ................                        [ 48%]
tests/plugins/test_mediaklikk.py ................                        [ 49%]
tests/plugins/test_mediavitrina.py .............................         [ 50%]
tests/plugins/test_mildom.py ........                                    [ 50%]
tests/plugins/test_mitele.py ..............                              [ 51%]
tests/plugins/test_mjunoon.py .........                                  [ 51%]
tests/plugins/test_mrtmk.py .........                                    [ 52%]
tests/plugins/test_n13tv.py ..............                               [ 52%]
tests/plugins/test_nbc.py .....                                          [ 52%]
tests/plugins/test_nbcnews.py ........                                   [ 53%]
tests/plugins/test_nbcsports.py .....                                    [ 53%]
tests/plugins/test_nhkworld.py .....                                     [ 53%]
tests/plugins/test_nicolive.py ........                                  [ 54%]
tests/plugins/test_nimotv.py .........                                   [ 54%]
tests/plugins/test_nos.py ............                                   [ 54%]
tests/plugins/test_nownews.py ..........                                 [ 55%]
tests/plugins/test_nrk.py .................                              [ 56%]
tests/plugins/test_ntv.py ........                                       [ 56%]
tests/plugins/test_okru.py ........                                      [ 56%]
tests/plugins/test_olympicchannel.py ................                    [ 57%]
tests/plugins/test_oneplusone.py ...........                             [ 57%]
tests/plugins/test_onetv.py ..............                               [ 58%]
tests/plugins/test_openrectv.py .......                                  [ 58%]
tests/plugins/test_orf_tvthek.py .....                                   [ 58%]
tests/plugins/test_picarto.py .........                                  [ 59%]
tests/plugins/test_piczel.py .....                                       [ 59%]
tests/plugins/test_pixiv.py .......                                      [ 59%]
tests/plugins/test_pluto.py ..........................................   [ 61%]
tests/plugins/test_pluzz.py ...........................                  [ 62%]
tests/plugins/test_qq.py ...............                                 [ 63%]
tests/plugins/test_radiko.py ..........                                  [ 63%]
tests/plugins/test_radionet.py ................                          [ 64%]
tests/plugins/test_raiplay.py ...........                                [ 64%]
tests/plugins/test_reuters.py ..........                                 [ 65%]
tests/plugins/test_rotana.py ......                                      [ 65%]
tests/plugins/test_rtbf.py ..........                                    [ 65%]
tests/plugins/test_rtpplay.py ................                           [ 66%]
tests/plugins/test_rtve.py .............                                 [ 66%]
tests/plugins/test_rtvs.py ..........                                    [ 67%]
tests/plugins/test_ruv.py .................                              [ 68%]
tests/plugins/test_sbscokr.py ......                                     [ 68%]
tests/plugins/test_schoolism.py .........                                [ 68%]
tests/plugins/test_senategov.py ......                                   [ 68%]
tests/plugins/test_showroom.py .............                             [ 69%]
tests/plugins/test_sportal.py .......                                    [ 69%]
tests/plugins/test_sportschau.py ......                                  [ 69%]
tests/plugins/test_ssh101.py ........                                    [ 70%]
tests/plugins/test_stadium.py .........                                  [ 70%]
tests/plugins/test_steam.py ..........                                   [ 71%]
tests/plugins/test_stream.py ........                                    [ 71%]
tests/plugins/test_streamable.py .....                                   [ 71%]
tests/plugins/test_streann.py ..............                             [ 72%]
tests/plugins/test_stv.py ......                                         [ 72%]
tests/plugins/test_svtplay.py .....................                      [ 73%]
tests/plugins/test_swisstxt.py ................                          [ 73%]
tests/plugins/test_teamliquid.py .............                           [ 74%]
tests/plugins/test_telefe.py ..........                                  [ 74%]
tests/plugins/test_tf1.py .............                                  [ 75%]
tests/plugins/test_tga.py .......                                        [ 75%]
tests/plugins/test_theplatform.py ......                                 [ 75%]
tests/plugins/test_tlctr.py .....                                        [ 76%]
tests/plugins/test_turkuvaz.py .................                         [ 76%]
tests/plugins/test_tv360.py .....                                        [ 77%]
tests/plugins/test_tv3cat.py ............                                [ 77%]
tests/plugins/test_tv4play.py ........                                   [ 77%]
tests/plugins/test_tv5monde.py ...........                               [ 78%]
tests/plugins/test_tv8.py .....                                          [ 78%]
tests/plugins/test_tv999.py ..........                                   [ 78%]
tests/plugins/test_tvibo.py .......                                      [ 79%]
tests/plugins/test_tvp.py ........                                       [ 79%]
tests/plugins/test_tvrby.py .........                                    [ 79%]
tests/plugins/test_tvrplus.py .........                                  [ 80%]
tests/plugins/test_tvtoya.py ...........                                 [ 80%]
tests/plugins/test_twitcasting.py ......                                 [ 80%]
tests/plugins/test_twitch.py ......................................      [ 82%]
tests/plugins/test_ustreamtv.py ............                             [ 82%]
tests/plugins/test_ustvnow.py .......                                    [ 83%]
tests/plugins/test_viasat.py ...........................                 [ 84%]
tests/plugins/test_vidio.py .........                                    [ 84%]
tests/plugins/test_vimeo.py ...........                                  [ 85%]
tests/plugins/test_vinhlongtv.py .......                                 [ 85%]
tests/plugins/test_viutv.py ......                                       [ 85%]
tests/plugins/test_vk.py .....................                           [ 86%]
tests/plugins/test_vlive.py .......                                      [ 86%]
tests/plugins/test_vrtbe.py ..........                                   [ 87%]
tests/plugins/test_vtvgo.py ................                             [ 87%]
tests/plugins/test_wasd.py .........                                     [ 88%]
tests/plugins/test_webcast_india_gov.py .......                          [ 88%]
tests/plugins/test_webtv.py ...............                              [ 89%]
tests/plugins/test_welt.py .................                             [ 89%]
tests/plugins/test_wwenetwork.py .....                                   [ 90%]
tests/plugins/test_youtube.py .......................................... [ 91%]
.                                                                        [ 91%]
tests/plugins/test_yupptv.py ......                                      [ 92%]
tests/plugins/test_zattoo.py ....................................        [ 93%]
tests/plugins/test_zdf_mediathek.py ...............                      [ 94%]
tests/plugins/test_zeenews.py ......                                     [ 94%]
tests/plugins/test_zengatv.py ..........                                 [ 94%]
tests/plugins/test_zhanqi.py .....                                       [ 95%]
tests/streams/test_dash.py ..................                            [ 95%]
tests/streams/test_dash_parser.py ....................                   [ 96%]
tests/streams/test_ffmpegmux.py .......................                  [ 97%]
tests/streams/test_hls.py ..............................                 [ 98%]
tests/streams/test_hls_filtered.py ......                                [ 99%]
tests/streams/test_hls_playlist.py ..                                    [ 99%]
tests/streams/test_stream_streamprocess.py .............                 [ 99%]
tests/streams/test_stream_to_url.py ........                             [ 99%]
tests/streams/test_stream_wrappers.py .                                  [100%]

=================================== FAILURES ===================================
_______________________ TestStreamToJSON.test_hls_stream _______________________

self = <tests.test_stream_json.TestStreamToJSON testMethod=test_hls_stream>

    def test_hls_stream(self):
        url = "http://test.se/stream.m3u8"
        master = "http://test.se/master.m3u8"

        stream = HLSStream(self.session, url, headers={"User-Agent": "Test"})
>       self.assertEqual(
            {
                "type": "hls",
                "url": url,
                "headers": {
                    "User-Agent": "Test",
                    "Accept": "*/*",
                    "Accept-Encoding": "gzip, deflate",
                    "Connection": "keep-alive",
                }
            },
            stream.__json__()
        )
E       AssertionError: {'typ[84 chars]ccept': '*/*', 'Accept-Encoding': 'gzip, defla[28 chars]ve'}} != {'typ[84 chars]ccept-Encoding': 'gzip, deflate, br', 'Accept'[32 chars]ve'}}
E         {'headers': {'Accept': '*/*',
E       -              'Accept-Encoding': 'gzip, deflate',
E       +              'Accept-Encoding': 'gzip, deflate, br',
E       ?                                               ++++
E
E                      'Connection': 'keep-alive',
E                      'User-Agent': 'Test'},
E          'type': 'hls',
E          'url': 'http://test.se/stream.m3u8'}

tests/test_stream_json.py:45: AssertionError
______________________ TestStreamToJSON.test_http_stream _______________________

self = <tests.test_stream_json.TestStreamToJSON testMethod=test_http_stream>

    def test_http_stream(self):
        url = "http://test.se/stream"
        stream = HTTPStream(self.session, url, headers={"User-Agent": "Test"})
>       self.assertEqual(
            {"type": "http",
             "url": url,
             "method": "GET",
             "body": None,
             "headers": {
                 "User-Agent": "Test",
                 "Accept": "*/*",
                 "Accept-Encoding": "gzip, deflate",
                 "Connection": "keep-alive",
             }},
            stream.__json__()
        )
E       AssertionError: {'typ[57 chars]T', 'body': None, 'headers': {'User-Agent': 'T[82 chars]ve'}} != {'typ[57 chars]T', 'headers': {'User-Agent': 'Test', 'Accept-[86 chars]None}
E         {'body': None,
E          'headers': {'Accept': '*/*',
E       -              'Accept-Encoding': 'gzip, deflate',
E       +              'Accept-Encoding': 'gzip, deflate, br',
E       ?                                               ++++
E
E                      'Connection': 'keep-alive',
E                      'User-Agent': 'Test'},
E          'method': 'GET',
E          'type': 'http',
E          'url': 'http://test.se/stream'}

tests/test_stream_json.py:26: AssertionError
=========================== short test summary info ============================
FAILED tests/test_stream_json.py::TestStreamToJSON::test_hls_stream - Asserti...
FAILED tests/test_stream_json.py::TestStreamToJSON::test_http_stream - Assert...
================= 2 failed, 2396 passed, 46 skipped in 56.79s ==================

[bastimeyer]

I'm currently trying to find out why that is being added to the header in the first place in my system (NixOS)

Post the result of pip list from the python env where streamlink is running.

When the brotli or brotlicffi packages are installed, urllib3.util.make_headers() inserts ',br' in the Accept-Encoding header and decodes br from the answers.

psf/requests#5783

but it would be nice if the test was more lenient with this kind of difference.

Yes, this needs to be fixed.

[bastimeyer]

psf/requests@5351469#diff-b4b6ac4e508ec3e142b8b8a3cf33b9ab925e3bdd03dbbe859384760746b1ac76R844

[iagocq]

The PR fixes the tests, thanks!

pip list output if you still need it

Package            Version
------------------ ------------
async-generator    1.10
async-timeout      3.0.1
atomicwrites       1.4.0
attrs              21.2.0
Brotli             1.0.9
brotlicffi         1.0.9.2
certifi            2021.5.30
cffi               1.14.6
chardet            4.0.0
charset-normalizer 2.0.1
curio              1.5
freezegun          1.1.0
idna               3.2
importlib-metadata 4.6.4
iniconfig          0.0.0
iso-639            0.4.5
iso3166            0.8
isodate            0.6.0
lxml               4.6.3
mock               4.0.3
more-itertools     8.8.0
outcome            1.1.0
packaging          20.9
pbr                5.6.0
pip                21.1.3
pluggy             1.0.0
py                 1.10.0
pycparser          2.20
pycryptodome       3.10.1
pyparsing          2.4.7
PySocks            1.7.1
pytest             6.2.5
python-dateutil    2.8.2
python-socks       1.2.4
requests           2.26.0
requests-mock      1.9.3
setuptools         57.2.0.post0
six                1.16.0
sniffio            1.2.0
sortedcontainers   2.4.0
toml               0.10.2
trio               0.19.0
urllib3            1.26.6
wcwidth            0.2.5
websocket-client   1.2.1
wheel              0.36.2
zipp               3.5.0