Accept OS Paths for File Marks

[pivotaljohn]

Problem Statement

ytt provides the user the ability to modify various metadata for files being processed through "File Marks".

A file mark targets a set of files described by a path. It then sets a piece of metadata (file type, whether to exclude entirely, whether to include in output) for all files matching that path.

The path value refers not to a filesystem path, but a "relative path". The "relative path" of a file are the path segments from the location specified in the --files/-f value to the location of the file.

Given this approach, it is easy to include two or more different files that end up having identical "relative path"es. When a File Mark is set for a

For example,

Given these inputs:

.
├── config
│   ├── custom
│   │   └── values.yml
│   ├── template
│   │   └── config.yml
│   └── values.yml
├── secrets
│   └── keys-to-the-kingdom.pem
└── custom
    └── values.yml

where:

• config/template/config.yml -- is a ytt template
• config/custom/values.yml -- contains a Data Values document (meant to customize config/values.yml)
• config/values.yml -- contains a Data Values document
• custom/values.yml -- contains YAML data intended to be data.load()'ed

In this case, we want to mark custom/values.yml as type=data. To do so, we first determine its relative path. This can be done with the --files-inspect flag:

$ ytt -f config/ -f custom/ --files-inspect
- custom/values.yml
- template/config.yml
- values.yml
- values.yml

And herein lies the problem: today, there's no way to differentiate between the Data Values files (config/values.yml and config/custom/values.yml) and the file targeted for loading (custom/values.yml).

Setting a file mark would affect both values.yml files:

$ ytt -f config/ -f custom/ --file-mark "values.yml:type=data"
ytt: Error:
- struct has no .foo field or method
    in <toplevel>
      template/config.yml:3 | foo: #@ data.values.foo

What's needed is the ability to set file marks using a path notation that can disambiguate when two files happen to have the same relative path.

Consideration: Windows Paths

Whatever solution must also support DOS/Windows-style paths...

C:\
├── config
│   ├── custom
│   │   └── values.yml
│   ├── template
│   │   └── config.yml
│   └── values.yml
├── secrets
    └── keys-to-the-kingdom.pem
W:\
└── custom
    └── values.yml

C:\>ytt -f c:\config\ -f w:\custom\ --file-mark "values.yml:type=data"

Consideration: Other Uses of Paths

Paths are used within templates as well (e.g. loading a module from a template, loading data from a file).

Ideally, any solution will make for a unified syntax for expressing paths.

In particular, it has already been established that a path beginning with a path separator is a root library-relative path.

Consideration: Difficulty in Detecting File Marking

Provide an easy way to determine what type a file is.

Possible Solutions

Filesystem Path

The user supplies a path value from the OS filesystem namespace sufficiently to disambiguate. At a minimum, this would be the "mount path" (i.e. the value given to the corresponding -f argument).
When that is insufficient to uniquely identify the desired target files, additional path-parts can be included.

Unix:

$ ytt -f config/ -f custom/ --file-mark "./custom/values.yml:type=data"

Note: os.SameFile() uses the inode and volume id (not the path expression) to determine whether two files are the same.
Notice that a file mark path of custom/values.yml would have continued to refer to two files: config/custom/values.yml and custom/values.yml. In this case, the standard "current directory" path-part is used to unambiguously refer to custom/values.yml

Windows:

C:\>ytt -f config\ -f w:\custom\ --file-mark "w:\custom\values.yml:type=data"

Note:os.SameFile() uses a file id (not the path expression) to determine if two files are the same.

---

Edits:

• adjust driving use case to include an overlapping path within the given mount points to surface that case.

[cppforlife]

in this example:

ytt -f config/ -f custom/ --file-mark "custom/values.yml:type=data"

what would happen if config/ has custom/values.yml as well (ie config/custom/values.yml)?

[pivotaljohn]

Yeah, I thought about that... I should've included it as a specific case.

The user can disambiguate using an OS relative path:

ytt -f config/ -f custom/ --file-mark "./custom/values.yml:type=data"

This would not match on the "ytt library path" but would match on the "mount path"

[pivotaljohn]

Working through that helped clarify: what we're talking about is allowing OS paths for file marks. 👍🏼