Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assert observedGeneration is incremented in Status.Conditions. #1586

Merged
merged 22 commits into from Dec 19, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d5aebb9
Update helpers so they check conditions when they are up to date
dprotaso Dec 7, 2022
6873420
When updating Gateway status conditions should increment their observ…
dprotaso Dec 7, 2022
0e5ed4c
allow injection of controllerNames
dprotaso Dec 8, 2022
5658f1d
When updating GatewayClass the status conditions should increment the…
dprotaso Dec 8, 2022
3fbe62e
fix casing of t.Error/Fatal messages
dprotaso Dec 9, 2022
b58b45f
removed pasted link
dprotaso Dec 9, 2022
009e835
refactor some helpers to assert observedGeneration is bumped
dprotaso Dec 9, 2022
de2d9c5
add HTTPRoute observed generation tests
dprotaso Dec 9, 2022
db554da
use a unique port when adding a new listener
dprotaso Dec 9, 2022
6582754
increase test timeout to a minute
dprotaso Dec 9, 2022
c8f4c5d
use real backends
dprotaso Dec 9, 2022
546ad16
fail test is an unexpected parent ref appears
dprotaso Dec 16, 2022
0b9d064
Provide better error messages for failed assertions
dprotaso Dec 16, 2022
ec9c9ca
fix fixture name
dprotaso Dec 16, 2022
9441114
fix stale count logging
dprotaso Dec 16, 2022
f87815b
create a deep copy prior to mutation and updating
dprotaso Dec 16, 2022
3e97ba4
drop redundant assertions
dprotaso Dec 16, 2022
46fb65d
address linting
dprotaso Dec 16, 2022
41f9802
use assertion helper
dprotaso Dec 16, 2022
e2ca2e9
log only the parent ref name
dprotaso Dec 16, 2022
c73eadb
update godoc
dprotaso Dec 16, 2022
759598d
put the gateway class observed gen bump behind a support flag
dprotaso Dec 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
90 changes: 90 additions & 0 deletions conformance/tests/gateway-observed-generation-bump.go
@@ -0,0 +1,90 @@
/*
Copyright 2022 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package tests

import (
"context"
"testing"
"time"

"github.com/stretchr/testify/require"
"k8s.io/apimachinery/pkg/types"

"sigs.k8s.io/gateway-api/apis/v1beta1"
"sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
"sigs.k8s.io/gateway-api/conformance/utils/suite"
)

func init() {
ConformanceTests = append(ConformanceTests, GatewayObservedGenerationBump)
}

var GatewayObservedGenerationBump = suite.ConformanceTest{
ShortName: "GatewayObservedGenerationBump",
Description: "A Gateway in the gateway-conformance-infra namespace should update the observedGeneration in all of it's Status.Conditions after an update to the spec",
Manifests: []string{"tests/gateway-observed-generation-bump.yaml"},
Test: func(t *testing.T, s *suite.ConformanceTestSuite) {

gwNN := types.NamespacedName{Name: "gateway-observed-generation-bump", Namespace: "gateway-conformance-infra"}

t.Run("observedGeneration should increment", func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()

namespaces := []string{"gateway-conformance-infra"}
kubernetes.NamespacesMustBeAccepted(t, s.Client, s.TimeoutConfig, namespaces)

original := &v1beta1.Gateway{}
err := s.Client.Get(ctx, gwNN, original)
require.NoErrorf(t, err, "error getting Gateway: %v", err)

// Sanity check
kubernetes.GatewayMustHaveLatestConditions(t, original)

all := v1beta1.NamespacesFromAll

mutate := original.DeepCopy()

// mutate the Gateway Spec
mutate.Spec.Listeners = append(mutate.Spec.Listeners, v1beta1.Listener{
Name: "alternate",
Port: 8080,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the moment, contour will not mark this update as accepted, we don't support a port other than 80 or 443

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do something trivial here like you've done with gw class, or add an annotation etc.?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or maybe change an existing listeners allowedroutes section?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't support a port other than 80 or 443

This restriction is odd - the API doesn't specify any constraints on the port numbers

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well I may be misspeaking a little, but at the very least Contour doesn't let you configure multiple HTTP or HTTPS Listeners, only one port is allowed for each at the moment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created an issue to move this discussion off the PR - #1607

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is passing now on Contour since observed generation is set correctly, the status we get is:

status:
  addresses:
  - type: IPAddress
    value: 172.24.255.203
  conditions:
  - lastTransitionTime: "2022-12-16T19:15:41Z"
    message: Gateway is accepted
    observedGeneration: 2
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2022-12-16T19:15:41Z"
    message: Listeners are not valid
    observedGeneration: 2
    reason: ListenersNotValid
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2022-12-16T19:15:41Z"
      message: Only one HTTP port is supported
      observedGeneration: 2
      reason: PortUnavailable
      status: "False"
      type: Accepted
    - lastTransitionTime: "2022-12-16T19:15:41Z"
      message: Invalid listener, see other listener conditions for details
      observedGeneration: 2
      reason: Invalid
      status: "False"
      type: Programmed
    name: alternate
    supportedKinds: []
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2022-12-16T19:15:41Z"
      message: Valid listener
      observedGeneration: 2
      reason: Programmed
      status: "True"
      type: Programmed
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

The Gateway as a whole should still be Accepted=true since some part of the config is valid, per: https://gateway-api.sigs.k8s.io/geps/gep-1364/#accepted

observedGenerations are all correct, though the status/validity isn't all there

Protocol: v1beta1.HTTPProtocolType,
AllowedRoutes: &v1beta1.AllowedRoutes{
Namespaces: &v1beta1.RouteNamespaces{From: &all},
},
})

err = s.Client.Update(ctx, mutate)
require.NoErrorf(t, err, "error updating the Gateway: %v", err)

// Ensure the generation and observedGeneration sync up
kubernetes.NamespacesMustBeAccepted(t, s.Client, s.TimeoutConfig, namespaces)

updated := &v1beta1.Gateway{}
err = s.Client.Get(ctx, gwNN, updated)
require.NoErrorf(t, err, "error getting Gateway: %v", err)

// Sanity check
kubernetes.GatewayMustHaveLatestConditions(t, updated)

if original.Generation == updated.Generation {
dprotaso marked this conversation as resolved.
Show resolved Hide resolved
t.Errorf("Expected generation to change because of spec change - remained at %v", updated.Generation)
}
})
},
}
14 changes: 14 additions & 0 deletions conformance/tests/gateway-observed-generation-bump.yaml
@@ -0,0 +1,14 @@
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: gateway-observed-generation-bump
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any way to use an existing Gateway for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep tests isolated - I don't want to change the base gateway if other tests assume it's static.

namespace: gateway-conformance-infra
spec:
gatewayClassName: "{GATEWAY_CLASS_NAME}"
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: All
78 changes: 78 additions & 0 deletions conformance/tests/gatewayclass-observed-generation-bump.go
@@ -0,0 +1,78 @@
/*
Copyright 2022 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package tests

import (
"context"
"testing"
"time"

"github.com/stretchr/testify/require"
"k8s.io/apimachinery/pkg/types"

"sigs.k8s.io/gateway-api/apis/v1beta1"
"sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
"sigs.k8s.io/gateway-api/conformance/utils/suite"
)

func init() {
ConformanceTests = append(ConformanceTests, GatewayClassObservedGenerationBump)
}

var GatewayClassObservedGenerationBump = suite.ConformanceTest{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GatewayClass is part of the environment provided by the person running the conformance tests. I personally don't think we should modify it. Maybe we can ensure that ObservedGeneration in status of the provided GatewayClass matches the generation of the resource though?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about this in the community sync: @youngnick made the point that we might as well do this now as it's something we'll need in the future if/when GatewayClass gains even more spec.

I'm personally +1 on Nick's point, I think we should keep this. However if this is a sticking point, I can live with us dropping it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can ensure that ObservedGeneration in status of the provided GatewayClass matches the generation of the resource though?

We do this already - this test is to ensure the observedGeneration changes when the spec changes

ShortName: "GatewayClassObservedGenerationBump",
Description: "A GatewayClass should update the observedGeneration in all of it's Status.Conditions after an update to the spec",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently Contour is not reconciling changes to GatewayClasses, just taking their state when they are first created/observed by the Contour Gateway API provisioner due to:

https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.GatewayClass

It is recommended that this resource be used as a template for Gateways. This means that a Gateway is based on the state of the GatewayClass at the time it was created and changes to the GatewayClass or associated parameters are not propagated down to existing Gateways. This recommendation is intended to limit the blast radius of changes to GatewayClass or associated parameters. If implementations choose to propagate GatewayClass changes to existing Gateways, that MUST be clearly documented by the implementation.

We haven't yet finished our thinking/implementation on this area so this fails for us

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that GatewayClass has an Accepted condition I would expect the observedGeneration of that condition to increment when the GWC spec changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah nvm, found where we were short circuiting the status update logic, this should work with Contour we have a small fix to make

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manifests: []string{"tests/gatewayclass-observed-generation-bump.yaml"},
Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
gwc := types.NamespacedName{Name: "gatewayclass-observed-generation-bump"}

t.Run("observedGeneration should increment", func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()

kubernetes.GWCMustBeAccepted(t, s.Client, s.TimeoutConfig, gwc.Name)

original := &v1beta1.GatewayClass{}
err := s.Client.Get(ctx, gwc, original)
require.NoErrorf(t, err, "error getting GatewayClass: %v", err)

// Sanity check
kubernetes.GatewayClassMustHaveLatestConditions(t, original)

mutate := original.DeepCopy()
desc := "new"
mutate.Spec.Description = &desc

err = s.Client.Update(ctx, mutate)
require.NoErrorf(t, err, "error updating the GatewayClass: %v", err)

// Ensure the generation and observedGeneration sync up
kubernetes.GWCMustBeAccepted(t, s.Client, s.TimeoutConfig, gwc.Name)

updated := &v1beta1.GatewayClass{}
err = s.Client.Get(ctx, gwc, updated)
require.NoErrorf(t, err, "error getting GatewayClass: %v", err)

// Sanity check
kubernetes.GatewayClassMustHaveLatestConditions(t, updated)

if original.Generation == updated.Generation {
t.Errorf("Expected generation to change because of spec change - remained at %v", updated.Generation)
}
})
},
}
7 changes: 7 additions & 0 deletions conformance/tests/gatewayclass-observed-generation-bump.yaml
@@ -0,0 +1,7 @@
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: gatewayclass-observed-generation-bump
spec:
controllerName: "{GATEWAY_CONTROLLER_NAME}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a significant change. As I understand it, this requires implementations to watch GatewayClasses matching their controller name and then update status on those. I think in some cases GatewayClass is part of the deployment model and each controller supports exactly one class. I could be wrong, but need to get some more input from other implementations before moving forward with this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can t.Skip() this test for now as not to hold the entire PR

Copy link
Contributor Author

@dprotaso dprotaso Dec 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put it behind a support feature flag for now (so that the manifests don't get applied)

description: "old"
85 changes: 85 additions & 0 deletions conformance/tests/httproute-observed-generation-bump.go
@@ -0,0 +1,85 @@
/*
Copyright 2022 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package tests

import (
"context"
"testing"
"time"

"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"

"sigs.k8s.io/gateway-api/apis/v1beta1"
"sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
"sigs.k8s.io/gateway-api/conformance/utils/suite"
)

func init() {
ConformanceTests = append(ConformanceTests, HTTPRouteObservedGenerationBump)
}

var HTTPRouteObservedGenerationBump = suite.ConformanceTest{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this sufficiently generic that the included code can work for every route type? This can be a follow up, just would be nice to have consistency across all Route types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do as a follow up

ShortName: "HTTPRouteObservedGenerationBump",
Description: "A HTTPRoute in the gateway-conformance-infra namespace should update the observedGeneration in all of it's Status.Conditions after an update to the spec",
Manifests: []string{"tests/httproute-observed-generation-bump.yaml"},
Test: func(t *testing.T, s *suite.ConformanceTestSuite) {

routeNN := types.NamespacedName{Name: "observed-generation-bump", Namespace: "gateway-conformance-infra"}
gwNN := types.NamespacedName{Name: "same-namespace", Namespace: "gateway-conformance-infra"}

acceptedCondition := metav1.Condition{
Type: string(v1beta1.RouteConditionAccepted),
Status: metav1.ConditionTrue,
Reason: "", // any reason
}

t.Run("observedGeneration should increment", func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()

namespaces := []string{"gateway-conformance-infra"}
kubernetes.NamespacesMustBeAccepted(t, s.Client, s.TimeoutConfig, namespaces)

original := &v1beta1.HTTPRoute{}
err := s.Client.Get(ctx, routeNN, original)
require.NoErrorf(t, err, "error getting HTTPRoute: %v", err)

// Sanity check
kubernetes.HTTPRouteMustHaveLatestConditions(t, original)

mutate := original.DeepCopy()
mutate.Spec.Rules[0].BackendRefs[0].Name = "infra-backend-v2"
err = s.Client.Update(ctx, mutate)
require.NoErrorf(t, err, "error updating the HTTPRoute: %v", err)

kubernetes.HTTPRouteMustHaveCondition(t, s.Client, s.TimeoutConfig, routeNN, gwNN, acceptedCondition)

updated := &v1beta1.HTTPRoute{}
err = s.Client.Get(ctx, routeNN, updated)
require.NoErrorf(t, err, "error getting Gateway: %v", err)

// Sanity check
kubernetes.HTTPRouteMustHaveLatestConditions(t, updated)

if original.Generation == updated.Generation {
t.Errorf("Expected generation to change because of spec change - remained at %v", updated.Generation)
}
})
},
}
12 changes: 12 additions & 0 deletions conformance/tests/httproute-observed-generation-bump.yaml
@@ -0,0 +1,12 @@
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: observed-generation-bump
namespace: gateway-conformance-infra
spec:
parentRefs:
- name: same-namespace
rules:
- backendRefs:
- name: infra-backend-v1
port: 8080
33 changes: 24 additions & 9 deletions conformance/utils/kubernetes/apply.go
Expand Up @@ -49,25 +49,31 @@ type Applier struct {
// four ValidUniqueListenerPorts.
// If empty or nil, ports are not modified.
ValidUniqueListenerPorts []v1beta1.PortNumber

// GatewayClass
GatewayClass string

// ControllerName
ControllerName string
}

// prepareGateway adjusts both listener ports and the gatewayClassName. It
// returns an index pointing to the next valid listener port.
func prepareGateway(t *testing.T, uObj *unstructured.Unstructured, gatewayClassName string, validListenerPorts []v1beta1.PortNumber, portIndex int) int {
err := unstructured.SetNestedField(uObj.Object, gatewayClassName, "spec", "gatewayClassName")
func (a Applier) prepareGateway(t *testing.T, uObj *unstructured.Unstructured, portIndex int) int {
err := unstructured.SetNestedField(uObj.Object, a.GatewayClass, "spec", "gatewayClassName")
require.NoErrorf(t, err, "error setting `spec.gatewayClassName` on %s Gateway resource", uObj.GetName())

if len(validListenerPorts) > 0 {
if len(a.ValidUniqueListenerPorts) > 0 {
listeners, _, err := unstructured.NestedSlice(uObj.Object, "spec", "listeners")
require.NoErrorf(t, err, "error getting `spec.listeners` on %s Gateway resource", uObj.GetName())

for i, uListener := range listeners {
require.Less(t, portIndex, len(validListenerPorts), "not enough unassigned valid ports for `spec.listeners[%d]` on %s Gateway resource", i, uObj.GetName())
require.Less(t, portIndex, len(a.ValidUniqueListenerPorts), "not enough unassigned valid ports for `spec.listeners[%d]` on %s Gateway resource", i, uObj.GetName())

listener, ok := uListener.(map[string]interface{})
require.Truef(t, ok, "unexpected type at `spec.listeners[%d]` on %s Gateway resource", i, uObj.GetName())

nextPort := validListenerPorts[portIndex]
nextPort := a.ValidUniqueListenerPorts[portIndex]
err = unstructured.SetNestedField(listener, int64(nextPort), "port")
require.NoErrorf(t, err, "error setting `spec.listeners[%d].port` on %s Gateway resource", i, uObj.GetName())

Expand All @@ -82,6 +88,12 @@ func prepareGateway(t *testing.T, uObj *unstructured.Unstructured, gatewayClassN
return portIndex
}

// prepareGatewayClass adjust the spec.controllerName on the resource
func (a Applier) prepareGatewayClass(t *testing.T, uObj *unstructured.Unstructured) {
err := unstructured.SetNestedField(uObj.Object, a.ControllerName, "spec", "controllerName")
require.NoErrorf(t, err, "error setting `spec.controllerName` on %s GatewayClass resource", uObj.GetName())
}

// prepareNamespace adjusts the Namespace labels.
func prepareNamespace(t *testing.T, uObj *unstructured.Unstructured, namespaceLabels map[string]string) {
labels, _, err := unstructured.NestedStringMap(uObj.Object, "metadata", "labels")
Expand All @@ -104,7 +116,7 @@ func prepareNamespace(t *testing.T, uObj *unstructured.Unstructured, namespaceLa

// prepareResources uses the options from an Applier to tweak resources given by
// a set of manifests.
func (a Applier) prepareResources(t *testing.T, decoder *yaml.YAMLOrJSONDecoder, gcName string) ([]unstructured.Unstructured, error) {
func (a Applier) prepareResources(t *testing.T, decoder *yaml.YAMLOrJSONDecoder) ([]unstructured.Unstructured, error) {
var resources []unstructured.Unstructured

// portIndex is incremented for each listener we see. For a manifest file
Expand All @@ -123,8 +135,11 @@ func (a Applier) prepareResources(t *testing.T, decoder *yaml.YAMLOrJSONDecoder,
continue
}

if uObj.GetKind() == "GatewayClass" {
a.prepareGatewayClass(t, &uObj)
}
if uObj.GetKind() == "Gateway" {
portIndex = prepareGateway(t, &uObj, gcName, a.ValidUniqueListenerPorts, portIndex)
portIndex = a.prepareGateway(t, &uObj, portIndex)
}

if uObj.GetKind() == "Namespace" && uObj.GetObjectKind().GroupVersionKind().Group == "" {
Expand Down Expand Up @@ -168,13 +183,13 @@ func (a Applier) MustApplyObjectsWithCleanup(t *testing.T, c client.Client, time
// MustApplyWithCleanup creates or updates Kubernetes resources defined with the
// provided YAML file and registers a cleanup function for resources it created.
// Note that this does not remove resources that already existed in the cluster.
func (a Applier) MustApplyWithCleanup(t *testing.T, c client.Client, timeoutConfig config.TimeoutConfig, location string, gcName string, cleanup bool) {
func (a Applier) MustApplyWithCleanup(t *testing.T, c client.Client, timeoutConfig config.TimeoutConfig, location string, cleanup bool) {
data, err := getContentsFromPathOrURL(location, timeoutConfig)
require.NoError(t, err)

decoder := yaml.NewYAMLOrJSONDecoder(data, 4096)

resources, err := a.prepareResources(t, decoder, gcName)
resources, err := a.prepareResources(t, decoder)
if err != nil {
t.Logf("manifest: %s", data.String())
require.NoErrorf(t, err, "error parsing manifest")
Expand Down