configdata.yml performance issues / switch away from PyYAML #2777
Comments
The-Compiler
added
config revolution
component: performance
|
This is still quite an issue, and I don't know of a solution as long as we're keeping the YAML file. However, I think [bindings.key_mappings]
default = {
"<Ctrl-[>": "<Escape>",
"<Ctrl-6>": "<Ctrl-^>",
"<Ctrl-M>": "<Return>",
"<Ctrl-J>": "<Return>",
"<Shift-Return>": "<Return>",
"<Enter>": "<Return>",
"<Shift-Enter>": "<Return>",
"<Ctrl-Enter>": "<Ctrl-Return>"
}
type_name = Dict
type_keytype = Key
type_valtype = Key
desc = This setting can be used to map keys to other keys.
When the key used as dictionary-key is pressed, the binding for the key used
as dictionary-value is invoked instead.
This is useful for global remappings of keys, for example to map Ctrl-[ to
Escape.
Note that when a key is bound (via `bindings.default` or
`bindings.commands`), the mapping is ignored.>>> import configparser
>>> cp = configparser.ConfigParser()
>>> cp.read('configdata.ini')
['configdata.ini']
>>> import json
>>> json.loads(cp['bindings.key_mappings']['default'])
{'<Ctrl-[>': '<Escape>', '<Ctrl-6>': '<Ctrl-^>', '<Ctrl-M>': '<Return>', '<Ctrl-J>': '<Return>', '<Shift-Return>': '<Return>', '<Enter>': '<Return>', '<Shift-Enter>': '<Return>', '<Ctrl-Enter>': '<Ctrl-Return>'}
>>> print(cp['bindings.key_mappings']['desc'])
This setting can be used to map keys to other keys.
When the key used as dictionary-key is pressed, the binding for the key used
as dictionary-value is invoked instead.
This is useful for global remappings of keys, for example to map Ctrl-[ to
Escape.
Note that when a key is bound (via `bindings.default` or
`bindings.commands`), the mapping is ignored.It requires a bit more parsing on our side (decoding |
|
I still hate the idea of making this an Looks like the following environments have C extensions:
And the following don't:
However, they took <3s to load the |
|
The master branch now shows a warning when it take longer than 1s (or 3s on the CI), pointing to this issue. Let's leave it at that for v1.0.0 for now and see if it's a problem in practice. Also, all environments on Travis now have C extensions for PyYAML available. |
|
In #3177 (comment) apparently @lejar has seen this while running the testsuite. @lejar Do you see this when starting qutebrowser as well, and does it actually take 14+ seconds? What if you start it via |
|
@The-Compiler Nope when I run it normally it is pretty snappy (start < 2 sec). Do note though that I've been doing all of this in a vmware image of ubuntu on a windows 10 host, with the vm being stored on a 7200 rpm hdd (though I'd hope that most of what the tests need is already in ram). It has 4 cores and 4gb ram. I ran the tests again with htop open and by the time the tests get to the yaml errors, it's using about 13.5% ram and 100% of one core. On the windows (host) side I'm monitoring disk IO but it is really minimal. One thing I did notice is that the reported failure duration increases with every failure: |
|
Hm, that is a very interesting observation! Can you please pastebin the full output you get (e.g. via |
|
I killed it after a couple hours. tox.log |
|
@lejar I've taken a first look at the log but I haven't figured out what's going on yet. I plan to get back to it somewhen this weekend though. |
|
I got this warning: edited by @The-Compiler to add proper code tags |
|
I think I finally tracked this down. It only affects the testsuite and shouldn't have affected real usage. |
|
Got a message telling me to report this issue here. Running Qutebrowser 1.3.3 on FreeBSD. |
|
Reopening as this is still an issue for some people. Probably time to switch to TOML or JSON. Maybe we might want to get rid of PyYAML entirely seeing that it repeatedly had issues with its maintainance so far. |
The-Compiler
changed the title
|
I would vote for json as well, if we pass some flags before dumping it to a file, it should be as easy to read in plaintext as well. With it being in core, we can count on it being fast everywhere (I hope...). @hjek way too late for a response, but if you change build options to build with |
|
Dumping some more thoughts here.
|
|
Florian Bruhin writes:
- Even YAML as a format really sucks
YAML is an extremely unfriendly format, I agree (eg: failing when there are
any tabs in the file whatsoever).
I would personally like either json or xml, since they seem to be in core
python so we won't take another dependency. I don't think that these files
need comments (they are autogenerated anyway), but if we just need it for the
auto-generated warning we could get something similar by having a "comment"
key in the top level object.
TOML is a very nice format, but it would be very nice to avoid adding more
dependencies so we don't have this problem again.
Maybe we should try profiling both of them against the session/configdata.yml
files and see which one performs better, since they are both noticeable in
overall qutebrowser profiles.
|
I quickly talked to @jgkamat in IRC about this - I agree for |
It's used for systemd units so it's almost impossible to find GNU/Linux machine nowadays without some TOML on it. Given that python seems to be converging on pyproject.toml it'll become even more commonly used in the near future. |
|
@zabbal systemd uses something ini-like which is not TOML. Also, I was mainly talking about the |
|
While I still think this important, I don't think it should block v2.0.0. We'll need PyYAML as a dependency for some while even after #5359 is done, so this can be done at a later point as well (with then fully dropping PyYAML in v3.0.0). |
|
Any idea? Debian 11, the libyaml package is installed.. WARNING: YAML load took unusually long, please report this at #2777 |
|
You might need libyaml-dev installed (and then re-install PyYAML in the virtualenv) for it to build the C extensions |
|
Thanks.. I used the old fashion way (python3 setup.py build), because pip downloads the latest version (5.4.1) from PyPi.org and the tarball doesn't contain source code at all (so what?!). Download from pyyaml.org instead. |
With the current
new-configbranch,test_init_benchmarkintest_configdata.pytakes (median):Even if this is a travis only thing, 200ms+ on each start is quite a hard hit to take. I can think of a couple of solutions:
ruamel.yaml(Switch to ruamel.yaml #2745)configdata.ymlhas changed and stuff though.asciidoc2html.py. If it's unavailable, load the yml file and show warnings as per above. Kind of difficult to figure out whether data is outdated or not.The text was updated successfully, but these errors were encountered: