Which toml package to use in python?
See also: toml-lang and PEP 680
The verions of the packages tested in this report.
Version | |
---|---|
toml | 0.10.2 |
tomli/tomli_w | 2.0.1; tomli_w: 1.0.0 |
tomlkit | 0.11.8 |
pytomlpp | 1.0.13 |
rtoml | 0.9.0 |
qtoml | 0.3.1 |
How the package dumps None
value in python
Literally <package>.dumps(None)
Dumped value or error | |
---|---|
toml | 'NoneType' object is not iterable |
tomli/tomli_w | 'NoneType' object has no attribute 'items' |
tomlkit | Expecting Mapping or TOML Container, <class 'NoneType'> given |
pytomlpp | dumps(): incompatible function arguments. The following argument types are supported: 1. (arg0: dict) -> str Invoked with: None |
rtoml | "null" |
qtoml | 'NoneType' object has no attribute 'items' |
How the package dumps key-value pair with value None
Literally <package>.dumps({"key": None})
Dumped value or error | |
---|---|
toml | |
tomli/tomli_w | Object of type <class 'NoneType'> is not TOML serializable |
tomlkit | Invalid type <class 'NoneType'> |
pytomlpp | cannot convert value None to proper toml type |
rtoml | key = "null" |
qtoml | TOML cannot encode None |
How the package dumps a list with None
value in it.
Literally <package>.dumps({"key": [1, 2, 3, None, 5]})
Dumped value or error | |
---|---|
toml | key = [ 1, 2, 3, "None", 5,] |
tomli/tomli_w | Object of type <class 'NoneType'> is not TOML serializable |
tomlkit | Invalid type <class 'NoneType'> |
pytomlpp | not a valid type for conversion None |
rtoml | key = [1, 2, 3, "null", 5] |
qtoml | bad type '<class 'NoneType'>' for dump_value |
How the package loads None
-like value in string
Literally <package>.loads('v1 = "null" v2 = "None"')
Loaded as | |
---|---|
toml | {'v1': 'null', 'v2': 'None'} |
tomli/tomli_w | {'v1': 'null', 'v2': 'None'} |
tomlkit | {'v1': 'null', 'v2': 'None'} |
pytomlpp | {'v1': 'null', 'v2': 'None'} |
rtoml | {'v1': 'null', 'v2': 'None'} |
qtoml | {'v1': 'null', 'v2': 'None'} |
How the package dumps a python dictionary with a heterogenous array.
Literally <package>.dumps({"v": [1, 1.2, True, "string"]})
Dumped value or error | |
---|---|
toml | v = [ 1, 1.2, true, "string",] |
tomli/tomli_w | v = [     1,     1.2,     true,     "string", ] |
tomlkit | v = [1, 1.2, true, "string"] |
pytomlpp | v = [ 1, 1.2, 1, 'string' ] |
rtoml | v = [1, 1.2, true, "string"] |
qtoml | v = [1, 1.2, true, 'string'] |
How the package loads a toml string with a heterogenous array.
Literally <package>.loads('v = [1, 1.2, True, "string"]')
Loaded as | |
---|---|
toml | Not a homogeneous array (line 2 column 1 char 1) |
tomli/tomli_w | {'v': [1, 1.2, True, 'string']} |
tomlkit | {'v': [1, 1.2, True, 'string']} |
pytomlpp | {'v': [1, 1.2, True, 'string']} |
rtoml | {'v': [1, 1.2, True, 'string']} |
qtoml | {'v': [1, 1.2, True, 'string']} |
How the package dumps a python dictionary with a nested array.
Literally <package>.dumps({"v": [[1], [1, 2]]})
Dumped value or error | |
---|---|
toml | v = [ [ 1,], [ 1, 2,],] |
tomli/tomli_w | v = [     [         1,     ],     [         1,         2,     ], ] |
tomlkit | v = [[1], [1, 2]] |
pytomlpp | v = [ [ 1 ], [ 1, 2 ] ] |
rtoml | v = [[1], [1, 2]] |
qtoml | v = [[1], [1, 2]] |
How the package loads a toml string with a nested array.
Literally <package>.loads('v = [[1], [1, 2]]')
Loaded as | |
---|---|
toml | {'v': [[1], [1, 2]]} |
tomli/tomli_w | {'v': [[1], [1, 2]]} |
tomlkit | {'v': [[1], [1, 2]]} |
pytomlpp | {'v': [[1], [1, 2]]} |
rtoml | {'v': [[1], [1, 2]]} |
qtoml | {'v': [[1], [1, 2]]} |
Whether the package preserves the order of the keys while dumps a python dictionary.
Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3})
yields a string
like c = 1\na = 2\nb = 3\n
.
Order kept? | |
---|---|
toml | Kept |
tomli/tomli_w | Kept |
tomlkit | Kept |
pytomlpp | Lost |
rtoml | Kept |
qtoml | Kept |
Whether the package preserves the order of the keys while loads a TOML string.
Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n')
yields
a dictionary with keys in the order of ['c', 'a', 'b']
.
Order kept? | |
---|---|
toml | Kept |
tomli/tomli_w | Kept |
tomlkit | Kept |
pytomlpp | Lost |
rtoml | Kept |
qtoml | Kept |
How the package dumps Unicode in python
Literally, <package>.dumps({"ä˝ ĺĄ˝": "世界"})
Dumped value | |
---|---|
toml | "ä˝ ĺĄ˝" = "世界" |
tomli/tomli_w | "ä˝ ĺĄ˝" = "世界" |
tomlkit | "ä˝ ĺĄ˝" = "世界" |
pytomlpp | 'ä˝ ĺĄ˝' = '世界' |
rtoml | "ä˝ ĺĄ˝" = "世界" |
qtoml | 'ä˝ ĺĄ˝' = '世界' |
How the package loads a file with unicode.
The file was created by:
## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
f.write('"ä˝ ĺĄ˝" = "世界"\n')
## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
loaded = <package>.load(f)
Loaded as | |
---|---|
toml | {'ä˝ ĺĄ˝': '世界'} |
tomli/tomli_w | File must be opened in binary mode, e.g. use open('foo.toml', 'rb') When loaded with rb :{'ä˝ ĺĄ˝': '世界'} |
tomlkit | {'ä˝ ĺĄ˝': '世界'} |
pytomlpp | {'ä˝ ĺĄ˝': '世界'} |
rtoml | {'ä˝ ĺĄ˝': '世界'} |
qtoml | {'ä˝ ĺĄ˝': '世界'} |
Test the compliance with the standard test suites for valid toml files here:
The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.
Result (toml-test v1.3.0) | |
---|---|
toml | key/escapes.toml Parsed as unexpected data. key/dotted.toml Found invalid character in key name: '"'. Try quoting the key name. (line 12 column 11 char 245) comment/tricky.toml Parsed as unexpected data. inline-table/key-dotted.toml Parsed as unexpected data. inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0) array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-string-table.toml list index out of range array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0) float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47) string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0) datetime/datetime.toml Parsed as unexpected data. datetime/local-time.toml Parsed as unexpected data. 86/100 (86.00%) passed |
tomli/tomli_w | string/escape-esc.toml Unescaped '' in a string (at line 1, column 10) 99/100 (99.00%) passed |
tomlkit | string/escape-esc.toml Invalid character 'e' in string at line 1 col 8 99/100 (99.00%) passed |
pytomlpp | string/escape-esc.toml Error while parsing string: escape sequence '\e' is not supported in TOML 1.0.0 and earlier (error occurred at line 1, column 9) 99/100 (99.00%) passed |
rtoml | string/escape-esc.toml invalid escape character in string: e at line 1 column 999/100 (99.00%) passed |
qtoml | comment/tricky.toml can't parse type (line 11, column 7) string/multiline-quotes.toml Didn't find expected newline (line 4, column 26) string/escape-esc.toml \e not a valid escape (line 1, column 33) datetime/milliseconds.toml Didn't find expected newline (line 2, column 27) datetime/datetime.toml Didn't find expected newline (line 2, column 18) 95/100 (95.00%) passed |
Test the compliance with the standard test suites for invalid toml files here:
Not OK
: The toml file is parsed without error, but expected to fail.OK
: All files are failed to parse, as expected. Showing the last parsing error.
Result (toml-test v1.3.0) | |
---|---|
toml | Not OK: key/start-dot.toml incorrectly parsed. Not OK: key/escape.toml incorrectly parsed. Not OK: key/special-character.toml incorrectly parsed. Not OK: key/two-equals3.toml incorrectly parsed. Not OK: key/two-equals2.toml incorrectly parsed. Not OK: inline-table/add.toml incorrectly parsed. Not OK: array/no-close-2.toml incorrectly parsed. Not OK: array/tables-1.toml incorrectly parsed. Not OK: array/no-close.toml incorrectly parsed. Not OK: array/extending-table.toml incorrectly parsed. Not OK: 39 more items incorrectly parsed. 188/237 (79.32%) passed |
tomli/tomli_w | OK: datetime/mday-under.toml Expected newline or end of document after a statement (at line 3, column 9) 237/237 (100%) passed |
tomlkit | Not OK: control/bare-cr.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed. 233/237 (98.31%) passed |
pytomlpp | Not OK: control/bare-cr.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. 235/237 (99.16%) passed |
rtoml | Not OK: control/bare-cr.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: integer/positive-hex.toml incorrectly parsed. Not OK: integer/positive-bin.toml incorrectly parsed. 232/237 (97.89%) passed |
qtoml | Not OK: inline-table/trailing-comma.toml incorrectly parsed. Not OK: inline-table/add.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. Not OK: control/comment-us.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/comment-lf.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/comment-null.toml incorrectly parsed. Not OK: table/duplicate-key-dotted-table.toml incorrectly parsed. Not OK: table/duplicate-key-dotted-table2.toml incorrectly parsed. Not OK: 2 more items incorrectly parsed. 225/237 (94.94%) passed |
Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:
https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid
The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.
Result (cpython tag 3.11.0) | |
---|---|
toml | apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49) five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97) dates-and-times/datetimes.toml Parsed as unexpected data. multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28) 8/12 (66.67%) passed |
tomli/tomli_w | OK, 12/12 (100%) passed |
tomlkit | OK, 12/12 (100%) passed |
pytomlpp | OK, 12/12 (100%) passed |
rtoml | OK, 12/12 (100%) passed |
qtoml | apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3) five-quotes.toml Didn't find expected newline (line 3, column 3) dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19) 9/12 (75.00%) passed |
Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:
https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid
Not OK
: The toml file is parsed without error, but expected to fail.OK
: All files are failed to parse, as expected. Showing the last parsing error.
Test the speed of loading and dumping the loaded
using data provided by pytomlpp
https://github.com/bobfang1992/pytomlpp/raw/master/benchmark/data.toml
Loading speed | Dumping speed | |
---|---|---|
toml | Excluded (heterogeneous arrays not supported) | Excluded (heterogeneous arrays not supported) |
tomli/tomli_w | 7.55s (1000 iterations) | 3.11s (1000 iterations) |
tomlkit | 150.98s (1000 iterations) | 2.14s (1000 iterations) |
pytomlpp | 1.41s (1000 iterations) | 0.98s (1000 iterations) |
rtoml | 0.98s (1000 iterations) | values must be emitted before tables |
qtoml | 18.39s (1000 iterations) | 4.75s (1000 iterations) |
Test the speed of loading and dumping the loaded using data
provided by rtoml
https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml
Loading speed | Dumping speed | |
---|---|---|
toml | Excluded (heterogeneous arrays not supported) | Excluded (heterogeneous arrays not supported) |
tomli/tomli_w | 1.56s (1000 iterations) | 0.55s (1000 iterations) |
tomlkit | 29.90s (1000 iterations) | 0.75s (1000 iterations) |
pytomlpp | 0.24s (1000 iterations) | 0.19s (1000 iterations) |
rtoml | 0.19s (1000 iterations) | 0.05s (1000 iterations) |
qtoml | 4.57s (1000 iterations) | 1.26s (1000 iterations) |
Test the speed of loading and dumping the loaded using data
provided by tomli
https://github.com/hukkin/tomli/raw/master/benchmark/data.toml
Loading speed | Dumping speed | |
---|---|---|
toml | Excluded (heterogeneous arrays not supported) | Excluded (heterogeneous arrays not supported) |
tomli/tomli_w | 1.11s (1000 iterations) | 0.38s (1000 iterations) |
tomlkit | 19.39s (1000 iterations) | 0.37s (1000 iterations) |
pytomlpp | 0.25s (1000 iterations) | 0.18s (1000 iterations) |
rtoml | 0.20s (1000 iterations) | 0.11s (1000 iterations) |
qtoml | 3.61s (1000 iterations) | 0.94s (1000 iterations) |
- Tests with
toml-test
v1.2.0 - Tests with
toml-test
v1.1.0 - Tests with
toml-test
v1.0.0 - Tests with python 3.11 (
tomllib
included)
pip install -U toml-bench
toml-bench
toml-bench --datadir /tmp/toml-bench
toml-bench --report ./README.md
toml-bench --comver 1.0.0
toml-bench --iter 5000
git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench