Prometheus remote-write

If you already run Prometheus, remote-write is the shortest path to getting those metrics into aiAxonIQ: it is a configuration change on the Prometheus side and nothing at all on your services.

Configure Prometheus

Add a remote_write block to prometheus.yml. Replace <base-endpoint> with the value Get Started shows in the dashboard — it differs between a hosted deployment and a self-hosted one, so no single URL is correct here:

remote_write:
  - url: <base-endpoint>/api/v1/prom/write
    headers:
      X-License-Key: oiq_4f3c2b1a9e8d7c6b5a4f3e2d1c0b9a8e

The endpoint is POST /api/v1/prom/write. Prometheus sends snappy-compressed protobuf by default, which is what this endpoint expects — no encoding options need setting.

A successful write returns 202 Accepted.

Your account must have metrics enabled. If metrics are not part of your plan, writes are rejected with 403 rather than silently dropped.

Limits per request

A remote-write request is rejected outright if it exceeds any of these. The defaults are generous enough that a normally-configured Prometheus will not approach them, but a large federated setup can:

• 8 MB compressed body
• 64 MB after decompression
• 10,000 timeseries
• 100,000 samples in total
• 128 labels per series
• 1,024 bytes per label name or value

Every series must carry a __name__ label. Prometheus always sets it; a custom client might not.

If you hit these, the fix is to shorten Prometheus's remote-write flush interval or reduce the batch size rather than to send fewer metrics — smaller, more frequent requests carry the same data.

Samples outside the accepted window

Samples are accepted if their timestamp falls within the last 30 days and no more than 1 hour in the future.

Samples outside that window are dropped silently — the request still succeeds with a 202 and the out-of-range samples simply do not appear. This is deliberate, so that one bad timestamp does not fail an otherwise good batch, but it does mean a backfill of older data will appear to succeed while writing nothing.

The future-dated allowance exists for clock skew between your Prometheus host and ingest. A host whose clock is more than an hour fast will lose every sample it sends, and the symptom is a healthy-looking Prometheus writing into a void — check clock sync before anything else if metrics from one host are missing entirely.

Rate limiting

Remote-write is subject to a per-account limit of 10,000 requests per minute, shared with your other ingest traffic. Exceeding it returns 429 with a Retry-After header.

Prometheus's remote-write client handles 429 responses by backing off and retrying, so a brief overshoot is absorbed without data loss.

Verifying

Prometheus exposes its own remote-write metrics, and they are the fastest way to tell whether the problem is on your side or ours:

# Samples successfully written.
rate(prometheus_remote_storage_samples_total[5m])

# Samples that failed permanently — should be flat at zero.
rate(prometheus_remote_storage_samples_failed_total[5m])

# How far behind the write path is falling.
prometheus_remote_storage_highest_timestamp_in_seconds
  - ignoring(remote_name, url) prometheus_remote_storage_queue_highest_sent_timestamp_seconds

A rising failure rate with a growing lag points at rejected requests — check the Prometheus server log, which records the status code and response body from the failed write.