Skip to main content
New: LLM Observability is now GA
aiAxonIQ
Try Playground
PricingUse cases
Get Started Free

Send data from Kubernetes

A DaemonSet collector for node and pod telemetry, where the license key goes, and the two collectors most clusters end up needing.

Before you start

aiAxonIQ has no Kubernetes agent of its own. It accepts OTLP, so the thing you install is the upstream OpenTelemetry Collector, pointed at your ingest endpoint. Everything below is stock chart configuration.

The shape most clusters need

There are two jobs, and they want different deployment modes:

  • Per-node telemetry — container logs, host metrics, and OTLP from workloads on that node. One collector per node: mode: daemonset.
  • Cluster-level telemetry — object counts, deployment replica status, events. There is exactly one of these per cluster, so it must not run per node: mode: deployment with replicaCount: 1.

Running the cluster receiver as a DaemonSet is the usual first mistake. Every node reports the same cluster-wide numbers, and the result is not wrong so much as multiplied by your node count.

Start with the DaemonSet. Add the second collector when you want workload status alongside the telemetry.

Store the key as a Secret

bash
kubectl create namespace observability

kubectl create secret generic observeiq \
  --namespace observability \
  --from-literal=license-key="$OIQ_LICENSE_KEY"

The key is an ingest credential for the whole workspace. It belongs in a Secret referenced by the collector, never in a values file committed to a repository.

Install the node collector

bash
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install observeiq-node open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values node-values.yaml

node-values.yaml:

yaml
mode: daemonset
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: observeiq
        key: license-key

presets:
  # Container logs from the node, with pod/namespace/container metadata attached.
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu: {}
        memory: {}
        disk: {}
        filesystem: {}
        network: {}

  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch:
      timeout: 5s
      send_batch_size: 1000

  exporters:
    otlphttp/observeiq:
      # The base URL shown on Get Started in the dashboard. A self-hosted
      # deployment behind the bundled nginx serves OTLP under /otlp; a hosted
      # one does not. Use the value the dashboard gives you.
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/observeiq]
      metrics:
        receivers: [otlp, hostmetrics]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/observeiq]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/observeiq]

Add the cluster collector

bash
helm install observeiq-cluster open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values cluster-values.yaml

cluster-values.yaml:

yaml
mode: deployment
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: observeiq
        key: license-key

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true

config:
  exporters:
    otlphttp/observeiq:
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip

The clusterMetrics preset is what populates the Kubernetes pages in the product — clusters, nodes, namespaces, pods and containers are all derived from k8s.* metrics. Without it those pages stay empty however much application telemetry is flowing, because they are describing the cluster rather than the workloads.

Point applications at the node collector

Inside the cluster, applications export to their own node's collector rather than to aiAxonIQ directly. The DaemonSet service address is stable:

yaml
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4318"
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

This keeps the license key in one place — the collector — instead of in every workload's environment, and it means the batching and retry behaviour is configured once.

Check it is working

bash
kubectl -n observability logs -l app.kubernetes.io/name=opentelemetry-collector --tail=50
kubectl -n observability port-forward svc/observeiq-node 8888:8888
curl -s localhost:8888/metrics | grep otelcol_exporter_send_failed

A rising otelcol_exporter_send_failed_* with 401 in the logs is a key that did not reach the pod; check the Secret name and the extraEnvs block. For everything else, Verify your data arrived walks the pipeline end to end.

Next steps