Dotted keys are not supported

[andrewhickman]

Status

• Parse dotted keys in tables
• Parse dotted keys in inline tables
• Preserve notion of dotted keys
• Dotted key ordering isn't always preserved #163
• Error when inserting dotted key into table
• Error when inserting table into dotted key

---

The following TOML file is not parsed successfully:

ssh.public-key-path = "C:/Users/andrew.hickman/.ssh/id_rsa.pub"

It fails with

TOML parse error at line 68, column 4
   |
68 | ssh.public-key-path = "C:/Users/andrew.hickman/.ssh/id_rsa.pub"
   |    ^
Unexpected `.`
Expected `=`

However the toml crate parses this correctly, and looking at the ABNF grammar I think it is a valid TOML file

toml = expression *( newline expression )

expression =/ ws keyval ws [ comment ]

dotted-key = simple-key 1*( dot-sep simple-key )

[andrewhickman]

Just spotted #73 and tried using that branch instead, and I can confirm that it parses this successfully

[ordian]

Support for toml 0.5 is tracked in #58.

[epage]

Some initial notes when researching this

Table-types summary

Tables

• Empty super tables are optional
• You cannot use a table to overwrite an item (value or table)
• Tables can be out-of-order

Dotted Keys

• Implicit tables can only merge with a previous implicit table from dotted key
• Table can be used to create child-table

Inline Tables

• Immutable

How toml_edit currently does it

Table

• Order-preserving map of Key to Item
• Create implicit tables for empty super tables

InlineTable

• Order-preserving map of Key to Item

Proposed API in #73

Dotted keys are just an Item to serve as a pointer to the appropriate table

• The table gets a has_dotted marker (unsure which implicit wasn't used)

Concerns

• Directly exposing the dotted-key pointer creates undefined behavior for users directly manipulating it

How tomlkit does it

Root Container appears to directly represent the document as-is

• A mapping is maintained between the direct representation and the users mental model
• Proxy types are used when the direct representation needs adjusting for a smooth API

Table and InlineTable

• independent Containers for their key-value pairs
• Implicit super tables are created for each level of a dotted key

Not quite sure how the rest of this is handled.

[epage]

The idea I'm playing with and trying to vet was briefly mentioned in #122, of having

pub struct Table {
    ...
    format: TableFormat,
}

pub enum TableFormat {
    ImplicitTable,
    DottedTable,
    InlineTable(...),
    Table(...),
}

• To retain dotted key ordering, we will add a position: Option<usize> to items
  • Changing a table's TableFormat will clear positions of all values (retaining insertion order) so that we don't end up with weird ordering when a dotted key turns into full Tables. and new items are inserted (problem case: first item would have position N and next would have position 0)
• Formatting information will be wrapped in a Option
  • Detect when formatting is undefined due to table-type change (ie changing types by users will make it None
  • Also allow detecting when users did not set any formatting
  • This will allow formatters to decide how they should format undefined formatting, whether naively or doing something fancier

TODO Finish filing this out

• What will Item and Value look like to support the above

[epage]

Some updates

• I was looking at merging table types
  • A TableFormat enum was being added along with structs for its invariants, like StandardTable and InlineTable
  • A table type could be left as unknown until formatting
  • Problems:
    • Led to complications when doing different operations that require knowing what the table type is, like sorting. Maybe that didn't have to be? I could just sort any ways
• I was looking into making the parser an iterator over Expression
  • Removes all the callbacks
  • Reduced the cosntant lookups because I kept the in-work table around
  • Problem: By not erroring inside of a parser, but outside, we lose automatically getting the position in the error message
  • Could still reduce the lookups by applying the same technique
  • Could still use expressions and have a shared callback

Overall, at this point, I am looking at keeping InlineTable and Table separate. There will be a way to mark either table as Dotted. I'll then add position information to TableKeyValuePari so we dotted positioning works like global table positioning.

I'm thinking of rolling out support gradually. First, we'll just parse the dotted tables without preserving their style and maybe not handling all error cases correctly. We'll then slowly add in those features.

[ordian]

Thanks for digging into this.

Something slightly related: you're probably familiar with #63 and table positioning. I think we should make to_string_in_original_order to_string, making the former the default Display behavior.

[epage]

I was thinking of that! Glad we are on the same page.

I've also wondered if we should switch away from formatting with Display. I think its behavior can surprise people (e.g. value.display() including decor when they probably expect the underlying value) and the interface makes things a bit limited (can't pass around state without ...Display types)

[epage]

(and yes, global table position was inspiring my table key positioning idea so we don't need markers)

[ordian]

I've also wondered if we should switch away from formatting with Display. I think its behavior can surprise people (e.g. value.display() including decor when they probably expect the underlying value) and the interface makes things a bit limited (can't pass around state without ...Display types)

Sure, we should only use Display when it makes sense. I think it does for the whole Document, but maybe not so for other types as you've mentioned.

[epage]

Random implementation notes from order preserving

• I initially tried storing the order in Key but it felt more natural in the parser to put it in the TableKeyValue
• I switched to TableKeyValue but we lose that information after we pass through get_children. So its back to being in the key

[epage]

I'm splitting #163 out and closing this