automatic state migrations

[ryanking]

It would be great if we could support automatically migrating state in terraform modules.

The use case is this– you ship a module foo with resources bar and baz. Later you decide to refactor baz into its own module, so now it would be foo containing bar and baz.this or something.

It would be great if the module could include a directive that would effectively run terraform state mv baz baz.this.

It would be really nice to have this integrated into terraform itself and I will raise an issue there at some point soon to start a discussion. In that mode, the migration would written in HCL and a first class part of the terraform model. It would be applied transiently when you run plan and persistently when you apply.

However, I recently thought of a cheaper / hacky way we could get this sooner with fogg.

We could add to our component makefile a task that does this–

1. after doing an init and get to make sure we have modules
2. read .terraform/modules/modules.json and parse out modules names and their local cached paths
3. at that cached path read a yaml file that includes migrations of the form x => y
4. for all x that exist, run terraform state mv x y

This task could prompt for confirmation before running and be set as a dependency for plan and apply. So when you run make apply and it discovers that there are unapplied migrations, it would prompt for confirmation, run the migrations and apply.

The major limitation of this is that you can't see a post-migration plan without altering the state file persistently, unless I am missing something.

[edulop91]

I think this is a great idea, would certainly save a lot of time. We could start asking for confirmation on each state mv to start out safe. Not too worried about applying state mv if we also ask for confirmation along the way. How would skip version tf module upgrades work? Is there a way to enforce/warn about this?

[ryanking]

Ok, amending this a bit.

plan takes a -state param, so what we can do is–

When we run make plan, first "plan" any state moves, then make a local copy of the state file (if we can), run the moves on the copy, then run plan. And try to make it clear to the user that we are doing this.

Then on apply we first detect if there are state moves. If so, ask user if we want to apply the moves. If yes, apply then continue to tf apply, otherwise abort.