Skip to content

Operator creates default collector sa even when one is specified #4529

@Marb0l

Description

@Marb0l

Component(s)

collector

What happened?

Description

Even if specified in the collector the operator will create a sa with the collector name and a clusterRole (with only /metrics permissions) but not a cluster role binding.

We are using pulumi to create a opentelemetry.io/v1beta1 resource where the serviceAccount is specified, we had to create it manually as by default the before described situation happens

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: metrics
  namespace: opentelemetry
  selfLink: >-
    /apis/opentelemetry.io/v1beta1/namespaces/opentelemetry/opentelemetrycollectors/metrics
spec:
  config:
    exporters:
      debug:
        verbosity: detailed
      prometheusremotewrite:
        auth:
          authenticator: oauth2client
        endpoint: >-
              'redacted'
        resource_to_telemetry_conversion:
          enabled: true
        tls:
          insecure_skip_verify: true
    extensions:
      oauth2client:
        client_id: >-
         "redacted"
        client_secret:  "redacted"
        token_url: >-
           "redacted"
    processors:
      resource:
        attributes:
        - action: upsert
          key: cluster
          value: predev-test
        - action: upsert
          key: opentelemetry
          value: "true"
        - action: delete
          key: k8s.pod.uid
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: otel-collector-self
            scrape_interval: 30s
            static_configs:
            - targets:
              - 127.0.0.1:8888
          - job_name: node-exporter-service
            metrics_path: /metrics
            scrape_interval: 30s
            static_configs:
            - targets:
              - >-
                kube-prometheus-stack-prometheus-node-exporter.prometheus.svc:9100
          - job_name: kubernetes-services
            kubernetes_sd_configs:
            - role: service
            relabel_configs:
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_probe
            - action: replace
              regex: (.+)
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_path
              target_label: __metrics_path__
            - action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              source_labels:
              - __address__
              - __meta_kubernetes_service_annotation_prometheus_io_port
              target_label: __address__
            - action: labelmap
              regex: __meta_kubernetes_service_label_(.+)
            - action: replace
              source_labels:
              - __meta_kubernetes_namespace
              target_label: kubernetes_namespace
            - action: replace
              source_labels:
              - __meta_kubernetes_service_name
              target_label: kubernetes_name
          - job_name: kubernetes-service-endpoints
            kubernetes_sd_configs:
            - role: endpoints
            relabel_configs:
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scrape
            - action: replace
              regex: (https?)
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scheme
              target_label: __scheme__
            - action: replace
              regex: (.+)
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_path
              target_label: __metrics_path__
            - action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              source_labels:
              - __address__
              - __meta_kubernetes_service_annotation_prometheus_io_port
              target_label: __address__
            - action: labelmap
              regex: __meta_kubernetes_service_label_(.+)
            - action: replace
              source_labels:
              - __meta_kubernetes_namespace
              target_label: kubernetes_namespace
            - action: replace
              source_labels:
              - __meta_kubernetes_service_name
              target_label: kubernetes_name
            - action: drop
              regex: node-exporter
              source_labels:
              - __meta_kubernetes_service_name
          - job_name: kubernetes-externalname-services
            kubernetes_sd_configs:
            - role: service
            relabel_configs:
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scrape
            - action: keep
              regex: true
              source_labels:
              - >-
                __meta_kubernetes_service_annotation_prometheus_monitor_resource_external_name_svc
            - action: replace
              regex: (https?)
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scheme
              target_label: __scheme__
            - action: replace
              regex: (.+)
              source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_path
              target_label: __metrics_path__
            - action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              source_labels:
              - __meta_kubernetes_service_external_name
              - __meta_kubernetes_service_annotation_prometheus_io_port
              target_label: __address__
            - action: replace
              regex: (.*)
              replacement: $1
              source_labels:
              - __meta_kubernetes_service_external_name
              target_label: instance
            - action: replace
              source_labels:
              - __meta_kubernetes_service_name
              target_label: service_name
            - action: replace
              regex: (.*)
              replacement: $1
              source_labels:
              - >-
                __meta_kubernetes_service_annotation_prometheus_custom_labels_instance_name
              target_label: instance_name
            - action: replace
              source_labels:
              - __meta_kubernetes_service_port_name
              target_label: job
            - action: labelmap
              regex: __meta_kubernetes_service_label_(.+)
          - job_name: kubernetes-pods
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_scrape
            - action: replace
              regex: (.+)
              source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_path
              target_label: __metrics_path__
            - action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              source_labels:
              - __address__
              - __meta_kubernetes_pod_annotation_prometheus_io_port
              target_label: __address__
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - action: replace
              source_labels:
              - __meta_kubernetes_namespace
              target_label: kubernetes_namespace
            - action: replace
              source_labels:
              - __meta_kubernetes_pod_name
              target_label: kubernetes_pod_name
          - honor_labels: true
            job_name: kubernetes-pushgateway
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_pushgateway
            - action: replace
              regex: (.+)
              source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_path
              target_label: __metrics_path__
            - action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              source_labels:
              - __address__
              - __meta_kubernetes_pod_annotation_prometheus_io_port
              target_label: __address__
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - action: replace
              source_labels:
              - __meta_kubernetes_namespace
              target_label: kubernetes_namespace
            - action: replace
              source_labels:
              - __meta_kubernetes_pod_name
              target_label: kubernetes_pod_name
    service:
      extensions:
      - oauth2client
      pipelines:
        metrics:
          exporters:
          - prometheusremotewrite
          - debug
          processors:
          - resource
          receivers:
          - prometheus
      telemetry:
        logs:
          development: true
          encoding: console
          initial_fields:
            service: metrics-collector
          level: debug
        metrics:
          address: 0.0.0.0:8888
          level: detailed
   ingress:
    route: {}
   mode: statefulset
  observability:
    metrics: {}
  podDnsConfig: {}
  replicas: 1
  resources: {}
  serviceAccount: my_custom_sa
  targetAllocator:
    allocationStrategy: consistent-hashing
    collectorNotReadyGracePeriod: 30s
    collectorTargetReloadInterval: 30s
    enabled: true
    filterStrategy: relabel-config
    observability:
      metrics: {}
    prometheusCR:
      enabled: true
      scrapeInterval: 30s
    replicas: 1
    resources: {}
    serviceAccount: my_custom_sa
  upgradeStrategy: automatic
status:
  image: >-
    'redacted'
  scale:
    replicas: 1
    selector: >-
      app.kubernetes.io/component=opentelemetry-collector,app.kubernetes.io/instance=opentelemetry.metrics,app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/name=metrics-collector,app.kubernetes.io/part-of=opentelemetry,app.kubernetes.io/version=latest
    statusReplicas: 1/1
  version: 0.126.0

Steps to Reproduce

Install the otel operator on a k8s cluster
create a collector for metrics with the sa specified

Expected Result

no additional resource will be created

Actual Result

a sa with the collector name and a role are created

Kubernetes Version

v1.32.9

Operator version

0.126.0

Collector version

0.126.0

Environment information

Environment

eks cluster, otel operator, pulumi

Log output

apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-collector
  namespace: opentelemetry
  selfLink: /api/v1/namespaces/opentelemetry/serviceaccounts/metrics-collector

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-operator-metrics
rules:
- nonResourceURLs:
  - /metrics
  verbs:
  - get

Additional context

I found the issue #2372 that reports almost the same problem but it's closed as solved

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions