Proposal for more granular file mappings per library

[jimmylewis]

Background

In the past, we've had numerous issues that highlight the limitations of file placement via libman: #407, #597, #702, and various others duped among those 3.

These can be summed up into a few high level issues:

1. The file structure upon installing is fixed based on how it appears within the provider's library, so users cannot exercise high fidelity control of how files are copied into their project (which is one of the primary goals of libman).
2. Users cannot configure a subset of files to one destination and another subset to a different destination.
3. Users want to copy a set of files into their project multiple times. Libman currently only allows one copy per library ID (e.g. name@version).

The proposal

This proposal introduces a new property in the manifest that allows more flexible file mappings, addressing all of the above issues. I've named it "fileMappings":

{
  "libraries": [
    {
      "library": "jquery@3.7.1",
      "provider": "jsdelivr",
      "fileMappings": [
        {
          "root": "dist/",
          "destination": "lib/jquery/",
          "files": [
            "*.min.js"
          ]
        }
      ]
    }
  ]
}

The fileMappings property is an array of mappings. There may be 0 or more.

Each mapping consists of 3 optional properties:

• root: the root of this mapping relative to the library root. This allows truncating a portion of the relative directory structure, so users can configure their file structure more carefully (addresses issue 1 listed above). This does not accept glob patterns.
  • If not specified, defaults to the library root (original folder structure is preserved)
• destination: where files identified by this mapping should be placed within the project.
  • If not specified, defaults to the library's configured installation path (i.e. respects destination on the library or the global defaultDestination). If that is not specified, it will be a validation error in the manifest.
• files: similar to the existing files property on the library, this is a list of file patterns (including glob patterns) relative to root that are to be included in this mapping.
  • If not specified, defaults to all files within the root.

Because there are any number of mappings allowed, users can now configure separate destinations for each set of files (addresses issue 2 above), and even configure the same file(s) to as many unique destinations as they desire (addresses issue 3 above).

Compatibility across libman versions

Because this is a purely additive change to the schema, there is no break to an existing libman.json file.

Pre-existing versions of the tooling will not be able to handle this new property. As such, a mixed environment could lead to a different state depending on which version of the tool is used. To mitigate this, the new property will require a new manifest version. This is to distinguish which versions of libman will support the new property.

1. For versions of libman that support this feature, the manifest will be invalid if it contains the fileMappings property but does not have a sufficiently high manifest version. This forces the manifest version to be increased.
2. For versions of libman that predate this feature, having a higher manifest version will be an error; this provides an indication that the older version is not compatible with the manifest, and will not work as desired. This covers the case where one user may upgrade the manifest, but their CI pipeline or teammate does not have the updated tool.
3. For older versions, if the manifest version is not updated and the fileMappings are set, they will not be respected. There isn't a way to address this without servicing older versions.

Points 1 and 2 ensure that the feature cannot be compatible on both new and old versions at the same time. However, point 3 is a case where outdated versions will not show errors even though the feature isn't supported.

The new manifest version is proposed to be "3.0" to match the current pending release of libman with that major version. This is timing sensitive - if a "3.0" version of libman releases without this feature, then we should bump the version again to 4.0 so that a 3.0 version doesn't exist without supporting this feature.

Edge cases

The existing properties are inherted into file mappings as appropriate:

• When destination is set on the library, it is the default for all fileMappings, unless they explicitly provide another.
• When files is omitted on the library and no fileMappings exist, all files are copied to the destination (no change)
• When files is omitted on the library and any fileMappings exist, only the files in the mapping are copied to their respective destinations.
• When files is set on the library, it is considered a shorthand for a fileMapping with an empty root (i.e. maintaining the same directory structure). I.e. these two are equivalent:

  {
    "libraries": [
      {
        "library": "jquery@3.7.1",
        "provider": "jsdelivr",
        "files": [ "*.min.js" ]
      },
      {
        "library": "jquery@3.7.1",
        "provider": "jsdelivr",
        "fileMappings": [
          {
            "files": [ "*.min.js" ]
          }
        ]
      }
    ]
  }
  

Alternatives considered

As this is a fairly verbose schema for each mapping, I did consider some other potentially simpler schemas first. However, none of them seemed to satisfy the scenarios robustly.

From #702, a proposal was made to map destination directories to file filters, e.g.

{
  "libraries": [
    {
      "library": "fileMappedTwiceExample",
      "mapping": {
        "**/foo.js": "somewhereElse/"
      }
    }
  ]
}

As discussed in that thread, there were ambiguities in how to handle glob patterns, and did not allow duplicate deployments of the same file filter to multiple destinations.

The thread also proposed a file filter mapping to an array of destinations:

"fileMapping": {
	"files/foo.js": [
		"somewhere/",
		"somewhereElse/"
	]
}

but again I personally feel like this is not as intuitive or flexible.

In #407, there was a proposal to expand files to support an object structure or string in its array. This leads to complex parsing (such as discriminated union support, which is a mess in C#). Most of the examples in that case were for individual files, and it has similar issues in supporting globs. The proposed object type (root/destination/files) could be done in this manner, but the complexity in parsing makes this a costlier option to implement.