Support full fidelity round-tripping of Pulumi.[yaml|json] #423
Comments
ellismg
added
area/cli
|
Seemingly relevant: go-yaml/yaml#132. |
|
Applies to |
|
I've looked into this and can't find a YAML parser that supports full-fidelity round-tripping. It appears that the
With this in mind, we should at least be able to preserve the comments in the config files, but things may still move around a bit. This would also require upgrading from v2 to v3 of this package. |
|
Have made good progress on this issue since my last update. After some more experimentation, we determined that the I have a proof of concept project working locally that can unmarshal a Pulumi config, add (or delete) a config value, and then marshal it back to YAML.
The next steps are to update the YAML unmarshaling/marshaling code in the pulumi CLI to use this package. |
Some YAML parsers don't correctly handle Byte-order Marks, so automatically strip it off during load. Related to #423
Some YAML parsers don't correctly handle Byte-order marks, so automatically strip it off during load. Related to #423 Co-authored-by: Justin Van Patten <jvp@justinvp.com>
|
Quick update on status of this issue: This is currently marked as "blocked" because some unforeseen compatibility issues with the new YAML parser library required us to revert the PR to add round-tripping support. The work to fix these issues was preempted by some other engineering priorities, so it's constrained by engineering investment rather than a specific issue. The actual round-tripping work is basically done, but we need to do some more work to make sure that the YAML marshaling/unmarshaling is compatible with the existing library. Once that is done, we need to swap out the libraries again and then merge the in-progress PR. |
|
The funny side effect of this is that https://app.pulumi.com/ complains that the git checkout used for a preview was dirty: Given that this preview ran from a CI job and that we have comments in our stack configs, I assume that the complaint is a result of overwriting the stack config. |
|
So, I'm gonna throw a radical idea out there: But has Pulumi considered simply not touching configuration files unless explicitly asked? I talk a little bit about this odd behaviour in this ticket and also here. Pulumi seems to expect users to manage the config files, but also seems to want to treat it as an extension of its internally managed state store. Is there any way Pulumi can just be a reader of files in source control and nothing else? |
AaronFriel
removed
the
good-first-issue
|
Comment-preserving edits will be implemented in the next release of Pulumi. |
|
@atrauzzi Our users find When editing a file like so: encryptionsalt: v1:ThS5UPxP9qc=:v1:UZYAXF+ylaJ0rGhv:9OTvBnOEDFgxs7btjzSu+LZ470vLpg==
# 🔴 header comment
config:
foo:a: some-value # 🟠 comment after value
# 🟡 comment before key
foo:b: some-value
foo:c:
a: A
b: B
c:
- 1
- 2
- 3 # 🟢 comment in array
# 🔵 comment after array
foo:d:
secure: v1:T1ftqhY0hqr+EJK6:+jvd5PMecFx80tcavzuZY4tLatgIfoe/xR72GA== # 🟣 comment on secret
# 🟥 footer commentAfter running encryptionsalt: v1:ThS5UPxP9qc=:v1:UZYAXF+ylaJ0rGhv:9OTvBnOEDFgxs7btjzSu+LZ470vLpg==
# 🔴 header comment
config:
foo:a: some-value # 🟠 comment after value
# 🟡 comment before key
foo:b: some-value
foo:c:
a: A
b: B
c:
- 1
- 2
- three # 🟢 comment in array
# 🔵 comment after array
foo:d:
secure: v1:T1ftqhY0hqr+EJK6:+jvd5PMecFx80tcavzuZY4tLatgIfoe/xR72GA== # 🟣 comment on secret
foo:e: E
# 🟥 footer comment |
11456: Implement YAML roundtripping, comment preserving edits r=AaronFriel a=AaronFriel By caching a copy of the raw bytes of the original loaded document - project, policy project, or stack - on save we: * Unmarshal the original bytes to a YAML AST * Recursively edit nodes of the document to match the values of the value we're saving, preserving trivia * Marshal the modified AST to []byte Closes #423, as we don't support comments in JSON presently (not using "JSON5") Closes #5235 Simplifies work done in #10797 and #10437 which used the previous `yamlutil` package to edit AST nodes directly. # Automation API Integration with automation API requires that operations either perform a "read-modify-write" in a single operation, or use a side channel to allow subsequent "read" (select stack?) and "write" operations to keep state of the original file. However I don't expect this to be needed. We don't have a reason to believe automation API users maintain comments in particular stack files, and doing so is already unsupported in automation scenarios. The Pulumi Service does not persist raw config files, thus `pulumi config refresh` and `CreateOrSelectStack` create config files without comments. n.b.: If you test this, due to the new behavior of this PR, `pulumi config refresh` will persist comments if there is an existing config file. I would expect automation API using local file workspaces to behave similarly. When using these tools to create a config file from the last deployment's definition, the file is created restored without comments. # Tradeoffs I believe that this closes #423 in spirit, but doesn't strictly implement round-tripping. There are a couple reasons for this. ## Marshaller interface and round-tripping simplification decisions We support unmarshaling "simplified" YAML, including allowing: ```yaml runtime: yaml # this unmarshals to a struct, not a scalar value config: someKey: "someScalar" # likewise ``` These simplifications make it much more difficult, the `Marshal` operation would _also_ need context of decisions made when unmarshaling to ensure a bijection. Otherwise we may save one of these as the other: ```yaml runtime: name: yaml ``` ```yaml runtime: yaml ``` This could be considered a feature, as it enforces a style guide - the decisions implemented in our custom `Marshaller` implementations become the style enforced on save. That said, I believe the new `Edit` API while not round-trippable, does reach a fixed point after one save through the new API. Once saved with this API, subsequent loads & saves without changes are idempotent. ## Indentation The YAML library uses a global indentation when marshaling values to YAML, as opposed to indentation being a property of AST nodes. This means we may lose other indentation. Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
|
The conversation in this collection of linked issues seems to read that anchors are supported in But, I tried using them: # Pulumi.yaml
name: my-app
runtime: yaml
description: my app
# template
tags: &defaultTags
Environment: ${pulumi.stack}
CreatedBy: pulumi
resources:
####################################
# L O G G R O U P S
# --------------------------------
# https://www.pulumi.com/registry/packages/aws/api- docs/cloudwatch/loggroup/
####################################
# template for log groups
logGroupTemplate: &logGroupTemplate
type: aws:cloudwatch:LogGroup
properties:
tags:
<<: *defaultTags
logGroupClass: STANDARD
retentionInDays: 731
skipDestroy: True # leave logs there in case we want to go back to them. They'll fall off after the retention period
# log group for lambda admin auth
lg_lambda_admin_auth:
<<: *logGroupTemplateand am getting Is it supposed to work or do I need to submit a new feature request? |
|
@jamest-pin Sorry for the delay in replying! This issue was closed and often closed issues don't get a lot of attention. You may want to open a new issue - it looks like this would belong on github.com/pulumi/pulumi-yaml. |
When writing information to the
Pulumi.yamlfile, we end up serializing back the entire structure and that serialization works by serializing each field in thepack.Packagein the order it was defined in the go object.In the case of
YAMLany comments would also be removed from the file.This means if we had a Pulumi.yaml file that looked like this:
If we read and save it it would look like this:
Ideally, we would have parsers that would turn these files not into raw go objects but some DOM that we could round trip with full fidelity.
The text was updated successfully, but these errors were encountered: