Full Other-Definitions support

[jpc4242]

This question is slightly related to #51.
Let's start this time as a question and then I'll create an issue depending on the response.

I don't know Go enough yet to be 100% sure, but reading the code it looks like currently:

• Other-Definitions from TOSCA 1.3 is parsed (in meta.go)
• But then it is completely ignored. I.e.: parsing or compiling a CSAR with Other-Definitions will ignore the files pointed by that entry.

If someone wants to use puccini just to validate, the algorithm is easy and it could even be done with an external script wrapper: parse normally then for each file in Other-Definitions parse an equivalent CSAR where you replace the value of Entry-Definitions. (Obviously if puccini implements this internally it would be better than a wrapper)

But my question here is about what happens after validation. Would the clout format support the result of compiling a CSAR containing multiple "toplevels"?

If that's not possible (or not easy), a "second best" could be to offer new arguments that allow to still compile a single toplevel but decide which one. E.g. puccini compile --other_def=3 would process the 3rd file in the Other-Definitions entry instead of the file in Entry-Definitions (and give an error if Other-Definitions is not present or has less than 3 entries)

[tliron]

Thanks for looking into this. I think you're on the right path here -- the "other definitions" are designed exactly for (potential) substitution mapping. In my view they should not all be joined into a single Clout, because they truly represent different topologies, and service composition is not a simple thing, basically out of scope for Puccini. (I discuss this in the FAQ.)

So I do think the right way would to output multiple Clouts. This could be done in various ways:

1. As you suggested, perhaps running Puccini multiple times and allowing the selection of which template to use in the CSAR. It's nice and easy to understand, and makes it easy to chain to other tools.
2. Compile all templates and output to a directory, rather than a file, in which each output would be its own Clout.
3. YAML of course supports multiple documents in one stream (separated by ---) so it's possible to have all Clouts in a single file. But this is only possible with YAML output, not JSON, CBOR, etc.

I think it can be possible to support all these options. Here are my current thoughts:

1. The default behavior should be as is it now, just compiling the "entry definitions".
2. A --select flag can be added to choose one of the "other definitions". We can support both a numerical index or refer to them by name.
3. A --all flag to compile all entries. When in "all" mode the output changes. Puccini will no longer output the Clout to stdout and instead require you to use the --output flag. Its meaning in this case will be a directory name, not a file name. How to name these files, I'm not sure... perhaps just a running sequence (0.yaml, 1.yaml, etc.). This will respect the --format flag so they can also be in JSON, CBOR, etc.
4. A --combine flag. This only works for YAML format. It will output all Clouts into a single YAML file, with the individual YAML documents separated by ---.

[jpc4242]

I like all options. My only doubt is with using names in --select. Other-Defintions uses a list of paths, not just names. You could even have folder1/thename.yaml and folder2/thename.yaml. This forces the user to know the exact paths in advance, which makes it harder

What about having also a --show-selectable that doesn't parse any yaml file (only TOSCA.meta) and returns (for example) a json or yaml file contain the array of valid names?

Also I have the gut feeling that when using numeric indexes the number 0 should be reserved for Entry-Defintions itself (so 0.yaml with --all would always exist)

P.S.: I know very little Go. For the time being I can only contribute ideas but not pull requests. I hope this changes in the future

[tliron]

All good ideas! I think it may be possible to have full-path support. True that you might not know it, but you might not also know the right index. Having both options seems easy enough to implement.

I did do some work on this today -- I finally implemented proper parsing of the Other-Directives entry. By the way, the spec mentions that you can surround names in quotes but doesn't mention escaping quotes. I did implement escaping for completion. We should probably update the spec for TOSCA 2.0 to mention this.

[jpc4242]

And do not forget escaping carriage returns and line feeds. (TOSCA.meta is line oriented )

[tliron]

I don't think that would be possible, because that would require escaping the entire TOSCA.meta. In any case, can paths even have newlines?

[jpc4242]

can paths even have newlines?

Not in win AFAIK, but it is definitely possible in Unix:

[root@madparquea4 test1]# touch 'one
> two'
[root@madparquea4 test1]# rm 'one
two'
rm: remove regular empty file ‘one\ntwo’? y

Of course possible does not mean advisable. But taking that into account can also prevent against obscure security attacks.
So I propose full escaping (but just in the file name, not the entire tosca.meta file. I.e the file above would appear as one\ntwo . And if the filename contains backlash (possible) then it is escaped as three\\four

[tliron]

I don't see how one can escape just the paths while still adhering to the spec of TOSCA.meta file. Even if there's a backslash before a newline, the TOSCA.meta processing would see that newline as the end of a line according to its parsing rules. I think this is an edge case that I'm OK with not supporting. Remember that the spec itself does not support escaping quotation marks at all.

[tliron]

The latest commit supports the --template flag for choosing one of the Other-Definitions. It can be either a path name or zero-based index, where "0" would be the Entry-Definitions, "1" would be the first Other-Definitions, etc.

I would appreciate if you give it a try! The scripts/test-csar can be a starting point.

[tliron]

Also, for finding out what Other-Definitions exist, there is a new puccini-tosca meta command.

[tliron]

Finally, the quickstart guide has been updated to briefly go over these new features.

[jpc4242]

Thank you, Tal. I've given word to some colleagues and they'll test this new feature.

[jpc4242]

All tests so far worked flawlessly

[tliron]

Great! I'm going to give more thought to how to handle exporting all compilation results in YAML. This would require some more significant refactoring. (By the way, it can be done quite fast because of intrinsic mulithreaded support in the library.)