the_workspace_run has erratic apply and destroy behaviour

[dejoost]

Terraform Cloud/Enterprise version

Terraform Cloud Plus

Terraform version

1.6.6

Terraform Configuration Files

resource "tfe_workspace_run" "runs" {
  for_each     = var.workspaces
  workspace_id = tfe_workspace.workspace[each.key].id

  dynamic "apply" {
    for_each = each.value.queue_all_runs == false ? toset([]) : toset([1]) # apply if queue_all_runs is null or true
    content {
      manual_confirm    = false
      wait_for_run      = true
      retry_attempts    = 5
      retry_backoff_min = 5
    }
  }

  destroy {
    manual_confirm    = false
    wait_for_run      = true
    retry_attempts    = 3
    retry_backoff_min = 10
  }

}

Debug Output

See screenshot

Expected Behavior

After removing the resource from the configuration the run settings should be removed from the workspace without altering the workspace resources

Actual Behavior

a destroy plan is queued in the workspace and apply is stuck for several minutes and end with an error message, see screenshot.

Additional Context

we applied tfe_workspace_run on around 80 existing workspaces, 2 succeeded (after a long time, +1 hour), 8 got stuck at applying and the other workspaces got queued, pending apply. All plans were successful. The queued and applying runs had to be cancelled through the run api.

When removing the runs on the 2 workspaces where it succeeded we observed this behaviour.