Skip to main content
New: LLM Observability is now GA

Troubleshooting

The pipeline stage by stage — what each failure looks like from outside, and which of the four places to look answers which question.

Before you start

Almost every "it isn't working" resolves to one of four stages. Establishing which one you are in takes about a minute and eliminates most of the list.

  1. The sender is not sending. No requests are leaving your service or collector.
  2. The request is being refused. Requests leave and come back with a 4xx.
  3. The request is accepted and the data is not where you are looking. 202, but nothing in the product.
  4. The data is there and something downstream is not using it. Records are queryable; a chart, alert or Kubernetes page is still empty.

Nothing is being sent

Signature: no exporter errors, no ingest activity, everything looks healthy.

An OpenTelemetry SDK that cannot export usually says so once at startup and then stays quiet. Turn on its diagnostic logging before assuming the network:

bash
export OTEL_LOG_LEVEL=debug

The three usual causes:

  • The instrumentation never loaded. In Node.js, importing the register hook from inside your entry file instead of --requireing it means the modules it patches are already resolved. The app starts and emits nothing.
  • OTEL_EXPORTER_OTLP_ENDPOINT is unset, so the SDK is exporting to localhost:4318 — its default — and failing silently against nothing.
  • The process exits before the batcher flushes. Short-lived jobs and tests need an explicit shutdown; without it the last batch is dropped, which is reliably the one you were watching for.

Requests are being refused

Signature: exporter logs carry a status code.

Each code means one thing, documented in full at Ingest endpoints and errors:

| Code | Meaning | First thing to check | |---|---|---| | 401 | Key missing, malformed, or revoked | That the header is X-License-Key and the value starts oiq_ | | 403 | Key valid, signal not enabled for the plan | Which signals your plan includes | | 404 | Wrong path | Whether the endpoint already ends in /v1 — the SDK appends it | | 413 | Batch over the size cap | Lower send_batch_size in the collector | | 429 | Rate limited | Whether many services share one key through a NAT gateway |

Two more that produce a 401 for reasons that are not about the key:

  • A revoked key still in a Secret. Revocation is immediate and not reversible; mint a new key rather than trying to restore one.
  • Whitespace from a shell heredoc or a Kubernetes Secret created with --from-file. A trailing newline is part of the value. --from-literal does not add one.

Accepted, but nothing shows up

Signature: 202 Accepted, empty screens.

Work through Verify your data arrived, which covers this case in full. The short list, in order of how often each turns out to be the answer:

  1. The time range. Narrow to the last 15 minutes before concluding anything. A dashboard on "last 24 hours" reads rolled-up tables that fill on their own schedule.
  2. A hardcoded timeUnixNano copied from an example. The record landed where that timestamp points — possibly outside retention, in which case it is accepted and never queryable.
  3. No service.name. The record is stored and appears under no service.
  4. The wrong workspace. A key belongs to one workspace; if you have several, check that the key and the dashboard you are looking at agree.

The data is there but something is not using it

Kubernetes pages are empty while application telemetry flows. Those pages are built from k8s.* metrics, which come from the cluster collector, not from your services. Enable the clusterMetrics preset — see Send data from Kubernetes.

An alert never fires. Check in this order: the rule is enabled; its window is long enough that a sparse signal produces at least one datapoint in it; and a notification channel is attached and passes its own test send. A rule with no channel evaluates correctly and tells nobody.

Synthetics show no results. Checks run from probe locations, and a fresh install has none registered until a probe is running. The checks are configured correctly and are not being executed by anything.

Logs are searchable but a field filter matches nothing. Field names come from the attributes you sent, and OTLP attribute keys are case-sensitive. service.name and Service.Name are different fields. Searching logs has the full field list.

When the collector is in the path

The collector's own metrics separate "my app reached the collector" from "the collector reached aiAxonIQ":

bash
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)'

Accepted rising while sent stays flat means the collector is receiving and failing to export — its logs carry the status code, and the table above applies. Nothing rising at all means your application is not reaching the collector, and the collector configuration is not the problem to look at yet.

Still stuck

Have these ready, because they are the first things anyone will ask for:

  • The base URL you are exporting to, exactly as configured.
  • The prefix of your license key — the first 12 characters, oiq_…. Never the whole key.
  • The status code and body of one failing request.
  • The service name and a UTC timestamp of something you sent and cannot find.

Next steps