Use Role and RoleBinding for namespace scoped operator failed

[FuyaoLi2017]

Hi Community,

I want to use a Role and RoleBinding to build a namespace scoped k8s operator. Only control the CR events in one namespace. If I change this to ClusterRole, ClusterRoleBinding. The problem will be gone. I just want to check if it is possible to make it Role and Rolebinding. My CRD is namespaced scoped.

I am getting the error

2022-02-17 00:01:24,203 ERROR org.apache.flink.kubernetes.operator.KubernetesOperatorEntrypoint [] - Exception occurred, but caught
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/flink.operator.io/v1/flinkapplications. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. flinkapplications.flink.operator.io is forbidden: User "system:serviceaccount:default:flink-native-k8s-operator" cannot list resource "flinkapplications" in API group "flink.operator.io" at the cluster scope.

This is my main class. Maybe I misused some APIs? Please suggest. Thanks!!!

  public static void main(String[] args) {

    try (KubernetesClient k8sClient = new DefaultKubernetesClient()) {
      String namespace = k8sClient.getNamespace();
      if (namespace == null) {
        LOG.info("No namespace found via config, assuming default.");
        namespace = "default";
      }

      LOG.info("Using namespace : " + namespace);

      final SharedInformerFactory informerFactory = k8sClient.informers();

      SharedIndexInformer<FlinkApplication> flinkAppinformer = informerFactory.sharedIndexInformerFor(FlinkApplication.class, 30 * 1000L);

      if (Constants.OPERATOR_WATCH_LEVEL.equalsIgnoreCase("namespace")) {
        flinkAppinformer = informerFactory
                .sharedIndexInformerFor(FlinkApplication.class, new OperationContext().withNamespace(namespace), 30 * 1000L);
      }

      LOG.info("Flink operator running in {} mode. In namespace mode, it will only watch {} namespace" +
                      " (the namespace operator is deployed) for Flink CR change events, " +
                      "In cluster mode, it will watch all namespaces' Flink CR change events.",
              Constants.OPERATOR_WATCH_LEVEL, namespace);

      MixedOperation<FlinkApplication, FlinkApplicationList, Resource<FlinkApplication>> flinkAppK8sClient
        = k8sClient.customResources(FlinkApplication.class, FlinkApplicationList.class);

      FlinkApplicationController flinkApplicationController = new FlinkApplicationController(
        k8sClient,
        flinkAppK8sClient,
        flinkAppinformer,
        namespace);

      flinkApplicationController.create();
      informerFactory.addSharedInformerEventListener(
        exception -> LOG.error("Exception occurred, but caught", exception));
      final Future<Void> startInformersFuture = informerFactory.startAllRegisteredInformers();
      startInformersFuture.get();

      flinkApplicationController.run();
    } catch (Exception exception) {
      LOG.error("Kubernetes Client Exception : ", exception);
    }
  }

[manusa]

If everything is namespaced, and your Role and RoleBiding are properly configured, everything should work.

I recall there might be some issues with some older versions of the Client and namespaced usage of SharedInformers (@shawkins)

[rohanKanojia]

You'll need to specify inNamepace in SharedInformerFactory in order to have namespaced informer:

SharedIndexInformer<FlinkApplication> flinkAppinformer = informerFactory.inNamespace("test").sharedIndexInformerFor(FlinkApplication.class, 30 * 1000L);

Or use more flexible DSL inform method:

SharedIndexInformer<FlinkApplication> flinkAppinformer = client.resources(FlinkApplication.class).inNamespace("test").inform(new ResourceEventHandler<FlinkApplication>() {
  @Override
  public void onAdd(FlinkApplication obj) {
    
  }

  @Override
  public void onUpdate(FlinkApplication oldObj, FlinkApplication newObj) {

  }

  @Override
  public void onDelete(FlinkApplication obj, boolean deletedFinalStateUnknown) {

  }
});

[FuyaoLi2017]

Hi @rohanKanojia @manusa , I am using Fabric8 version 5.5.0. The problem still persists after changing it to the syntax you suggested.

I tried to use the syntax suggested. The problem is still the same. I also suspect it is a problem with the kubernetes client. I tried to switch to namespaced client.

With the code below. I am stilling have trouble to use Role and RoleBinding. It seems this CRD is a cluster level resource. The service account I configured is Role/RoleBinding and it is not allowed to view the clusterLevel things.

I think the error

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/flink.operator.io/v1/flinkapplications. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. flinkapplications.flink.operator.io is forbidden: User "system:serviceaccount:default:flink-native-k8s-operator" cannot list resource "flinkapplications" in API group "flink.operator.io" at the cluster scope.

is thrown from the line below.
If I change the Role/RoleBinding below to ClusterRole/ClusterRoleBinding below. There will be no problem and Operator will only watch expected namespace.

informerFactory.addSharedInformerEventListener(
        exception -> LOG.error("Exception occurred, but caught", exception));

I updated the main class (but still the same error):

  public static void main(String[] args) {

    try (KubernetesClient k8sClient = Constants.OPERATOR_WATCH_LEVEL.equalsIgnoreCase("cluster")
            ? new DefaultKubernetesClient()
            : new DefaultKubernetesClient().inNamespace(Constants.FLINK_OPERATOR_RELEASE_NAMESPACE)) {
      String namespace = k8sClient.getNamespace();
      if (namespace == null) {
        LOG.info("No namespace found via config, assuming default.");
        namespace = "default";
      }

      LOG.info("Using namespace : " + namespace);

      final SharedInformerFactory informerFactory = k8sClient.informers();

      SharedIndexInformer<FlinkApplication> flinkAppinformer = informerFactory.sharedIndexInformerFor(FlinkApplication.class, 30 * 1000L);

      if (Constants.OPERATOR_WATCH_LEVEL.equalsIgnoreCase("namespace")) {
// adding or delete OperationContext doesn't influence the error result
        flinkAppinformer = informerFactory.inNamespace(Constants.FLINK_OPERATOR_RELEASE_NAMESPACE)
                .sharedIndexInformerFor(FlinkApplication.class, new OperationContext().withNamespace(namespace), 30 * 1000L);
      }

      LOG.info("Flink operator running in {} mode. In namespace mode, it will only watch {} namespace" +
                      " (the namespace operator is deployed) for Flink CR change events, " +
                      "In cluster mode, it will watch all namespaces' Flink CR change events.",
              Constants.OPERATOR_WATCH_LEVEL, namespace);

      MixedOperation<FlinkApplication, FlinkApplicationList, Resource<FlinkApplication>> flinkAppK8sClient
        = k8sClient.customResources(FlinkApplication.class, FlinkApplicationList.class);

      FlinkApplicationController flinkApplicationController = new FlinkApplicationController(
        k8sClient,
        flinkAppK8sClient,
        flinkAppinformer,
        namespace);

      flinkApplicationController.create();


      if (Constants.OPERATOR_WATCH_LEVEL.equalsIgnoreCase("namespace")) {
        informerFactory.inNamespace(Constants.FLINK_OPERATOR_RELEASE_NAMESPACE).addSharedInformerEventListener(
                exception -> LOG.error("During namespace mode initialization: Exception occurred, but caught", exception));
      } else {
        informerFactory.addSharedInformerEventListener(
                exception -> LOG.error("During cluster mode initialization: Exception occurred, but caught", exception));
      }

      final Future<Void> startInformersFuture = informerFactory.startAllRegisteredInformers();
      startInformersFuture.get();

      flinkApplicationController.run();
    } catch (Exception exception) {
      LOG.error("Kubernetes Client Exception : ", exception);
    }
  }

Role.yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flink-native-k8s-operator
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups:
      - flink-native-k8s-operator
    resources:
      - "*"
    verbs:
      - "*"
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - persistentvolumeclaims
      - events
      - configmaps
      - secrets
    verbs:
      - "*"
  - apiGroups:
      - apps
    resources:
      - deployments
      - replicasets
    verbs:
      - "*"
  - apiGroups:
      - extensions
    resources:
      - deployments
      - ingresses
    verbs:
      - "*"
  - apiGroups:
      - flink.operator.io
    resources:
      - flinkapplications
    verbs:
      - "*"
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - "*"

RoleBinding.yaml

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flink-native-k8s-operator-role-binding
  namespace: {{ .Release.Namespace }}
subjects:
  - kind: ServiceAccount
    name: flink-native-k8s-operator
    namespace: {{ .Release.Namespace }}
roleRef:
  kind: Role
  name: flink-native-k8s-operator
  apiGroup: rbac.authorization.k8s.io

serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: flink-native-k8s-operator
  namespace: {{ .Release.Namespace }}

[FuyaoLi2017]

I think this issue I meet here, is pretty similar to this problem.
#2331

So every POJO should implement Namespaced interface? Or just the FlinkApplication.

This is my definition for FlinkApplication.

public class FlinkApplication extends CustomResource<FlinkApplicationSpec, FlinkApplicationStatus> implements
    Namespaced

For FlinkApplicationList and FlinkApplicationSpec and FlinkApplicationStatus and JobStatus (Within FlinkApplicationStatus), I haven't add the Namespaced Interface. I checked the examples in https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/crds
I think there should be not problem with this one.

[FuyaoLi2017]

This is my FlinkApplication definition.

@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonDeserialize()
@EqualsAndHashCode(callSuper = true)
@Version(FlinkApplication.VERSION)
@Group(FlinkApplication.GROUP)
public class FlinkApplication extends CustomResource<FlinkApplicationSpec, FlinkApplicationStatus> implements
    Namespaced {

    public static final String GROUP = "flink.operator.io";
    public static final String VERSION = "v1";

    private FlinkApplicationSpec spec;
    private FlinkApplicationStatus status;

    public FlinkApplicationSpec getSpec() {
        return spec;
    }

    public void setSpec(FlinkApplicationSpec spec) {
        this.spec = spec;
    }

    public FlinkApplicationStatus getStatus() {
        return status;
    }

    public void setStatus(FlinkApplicationStatus status) {
        this.status = status;
    }

    @Override
    public String toString() {
        return "FlinkApplication{"+
                "apiVersion='" + getApiVersion() + "'" +
                ", metadata=" + getMetadata() +
                ", spec=" + spec +
                ", status=" + status +
                "}";
    }

    @Override
    public ObjectMeta getMetadata() {
        return super.getMetadata();
    }
}

[FuyaoLi2017]

Hi @manusa @rohanKanojia , could you share insights on this problem? Thanks!

[rohanKanojia]

With the code below. I am stilling have trouble to use Role and RoleBinding. It seems this CRD is a cluster level resource. The service account I configured is Role/RoleBinding and it is not allowed to view the clusterLevel things.

If your CustomResourceDefinition is of Cluster scope, you should be using ClusterRole/ClusterRoleBinding instead of Role/RoleBinding. As per Kubernetes documentation:

ClusterRoles have several uses. You can use a ClusterRole to:
define permissions on namespaced resources and be granted within individual namespace(s)
define permissions on namespaced resources and be granted across all namespaces
define permissions on cluster-scoped resources
If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

[rohanKanojia]

You're supposed to use Namespaced interface only when your CustomResourceDefinition is of Namespaced scope. If it's not, you don't need to add implements Namespaced to your CustomResource class

[FuyaoLi2017]

Thanks for the reply. Hi @rohanKanojia , my CRD is of namespaced scope, but I am still facing the problem. Do you see any problem with my main class code? Thanks.

[stale]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!