LLM serving observability from scratch

2026-07-05 · llm, vllm, observability, opentelemetry

An OpenAI-compatible endpoint looks the same whether it is healthy or seconds from falling over. I wanted to see the inside, so I put vLLM on my desktop’s RTX 5070 Ti, pointed Prometheus at its metrics, sent its OpenTelemetry traces to Jaeger, and pushed until the scheduler had to make decisions. Everything below is captured data from those runs, rendered from committed JSON, no dashboard screenshots. The code blocks are read from the actual files at build time, and the whole setup lives in scripts/vllm-observability if you want to run it yourself.

The stack

The observability side is three containers, no GPU needed, so rootless podman is enough.

compose.yaml 
services:
  prometheus:
    image: docker.io/prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-lifecycle
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro,Z

  grafana:
    image: docker.io/grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      # anonymous read-only so the dashboard is watchable from the LAN;
      # editing needs the admin login
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro,Z

  jaeger:
    image: docker.io/jaegertracing/jaeger:latest
    ports:
      - "16686:16686" # UI
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

vLLM runs natively on the host and Prometheus reaches it through podman’s host alias.

prometheus/prometheus.yml 
global:
  scrape_interval: 5s

scrape_configs:
  # 1s scrape so captured replays have per-second resolution
  - job_name: vllm
    metrics_path: /metrics
    scrape_interval: 1s
    static_configs:
      # vLLM runs natively on the host; host.containers.internal is podman's host alias
      - targets: ["host.containers.internal:8000"]

  - job_name: gpu
    scrape_interval: 1s
    static_configs:
      - targets: ["host.containers.internal:9835"]

One command serves the model, exposes /metrics, ships traces, and enables tool calls.

serve.sh 
#!/usr/bin/env bash
# vLLM with Prometheus metrics (built-in at /metrics) and OTel traces to Jaeger.
# The observability stack must be up first: podman compose up -d
set -euo pipefail
cd "$(dirname "$0")"

MODEL="${MODEL:-unsloth/qwen3.5-4b}"

# FlashInfer JIT-compiles kernels at startup; CUDA 13.2 rejects Fedora 44's
# gcc 16 as host compiler, so point nvcc at the gcc15 compat package.
export NVCC_PREPEND_FLAGS='-ccbin /usr/bin/g++-15'

# vLLM does not set an OTel service name; without this, traces land in
# Jaeger under "unknown_service".
export OTEL_SERVICE_NAME=vllm

exec uv run vllm serve "$MODEL" \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.7 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 4096 \
  --limit-mm-per-prompt '{"image": 0, "video": 0}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --otlp-traces-endpoint grpc://localhost:4317 \
  --collect-detailed-traces all \
  --port 8000

Getting an 8 GB model onto a 16 GB desktop card took three failed startups, and three of those flags are the scar tissue: the compiler override feeds FlashInfer’s kernel JIT,1 image inputs are off because profiling the vision encoder OOMs the card,2 and the memory target stays at 0.7 to leave room for the desktop itself.3 Without OTEL_SERVICE_NAME, traces land in Jaeger as unknown_service.

GPU metrics come from a 20-line nvidia-ml-py exporter instead of DCGM, which would want the GPU-in-container setup this machine does not have.

Serving 32 requests at once on one GPU

Every request being generated keeps its attention state in GPU memory, the KV cache, and that state grows by one entry per generated token. After the weights take their 8 GB, this card has room for about 33k tokens of it.

requests waiting queue admit running batch + + + + cache full → preempt rejoins later, recomputes one block per generated token KV cache · ~33k tokens
One decode step reads the weights once and advances every running request by one token, so each step also takes one block per request from the pool. The scheduler's whole job is deciding who runs when the pool cannot hold everyone.

So 64 requests at concurrency 32, each generating up to 1024 tokens, cannot fit. At full depth the batch wants 32 × 1k, which is the entire cache.

The requests themselves are nothing special, and that is deliberate. The load generator round-robins eight short programming questions as chat completions, each allowed up to 1024 tokens. These are the real responses, captured one at a time on an idle server:

the eight prompts and what came back

947 completion tokens · finished on its own

thinking

This is a great question about data structures, specifically hash maps and collision handling. Let me provide a comprehensive explanation of how hash maps handle collisions, covering the main strategies used.

I should cover:

  1. What are collisions in the context of hash maps
  2. Different collision handling strategies
  3. How each strategy works with examples
  4. Trade-offs between different approaches

Let me structure this clearly with examples in code would be helpful.

Understanding Hash Map Collisions

A hash map is a data structure that maps keys to values using a hash function. When multiple keys hash to the same index, a collision occurs. Efficient collision resolution is essential for the performance and functionality of hash maps.

What Is a Hash Collision?

A hash map assigns each key to a bucket using a hash function. If two different keys produce the same index, they collide. For example, if the hash function maps both “sauf” and “bob” to index 3, the hash map must decide how to store both keys under the same index.

Common Collision Resolution Techniques

1. Chaining

Each bucket holds a list or linked structure of elements with the same index.

  • How It Works:

    • When a key hashes to the same bucket as the next, it is added to a linked list or array.
    • Accessing a value involves hashing and then traversing the list.

  • Example:

    // Java-like structure
class HashEntry {
    Key key;
    Object value;
    HashEntry next;
}
Map(int capacity, () { return list.get(hash(key)); }

  • Performance:

    • Average case: O(1) lookup
    • Worst case: O(n) if all keys collide

  • Pros: Simple to implement, works in many scenarios.

  • Cons: Extra memory due to node structures, can degrade with poor hash functions.

2. Open Addressing

All elements are stored directly in the underlying array.

a) Linear Probing

  • Upon collision, search for the next free slot in the array.

  • Example: [null, keyA, keyB, keyA, keyC]

  • Pros: Better cache locality.

  • Cons: Produces clustering, limiting effectiveness.

b) Quadratic Probing

  • Uses the square of the index to find the next slot.

  • Formula: (hash + i²) % tableSize

  • Pros: Helps reduce clustering.

  • Cons: Still suffers from primary clustering; limited range.

c) Double Hashing

  • Uses a second hash function to compute probe steps.

  • Formula: (hash + i * h2(key)) % tableSize

  • Pros: Minimal clustering.

  • Cons: More complex to implement.

d) Treating Keys with Equality
  • All identical keys map to the same index with a single call.
  • Useful for sets when auto-detection is required.

3. Separate Chaining (Efficient Variants)

  • Uses arrays of lists at each index.
  • Often faster in modern implementations due to optimized memory usage.

Trade-offs Summary

TechniqueTime Complexity (Avg)Time Complexity (Worst)Space Overhead
ChainingO(1)O(n)Higher
Linear ProbingO(1)O(n)Low
Quadratic ProbingO(1)O(n)Low–Moderate
Double HashingO(1)O(n)Moderate

Summary

  • Chaining is flexible and manageable but consumes more memory.
  • Open Addressing uses less memory but can suffer from clustering.
  • Choice of technique depends on expected load, key distribution, and memory constraints.

Most modern hash map implementations use chaining, separate chaining, or variants of open addressing, optimizing based on performance, memory, and use-case requirements.

Understanding these techniques allows developers to make informed decisions when implementing or using hash maps in practice.

Five of the eight hit the 1024 token cap and the shortest still took 454 tokens. A reasoning model thinks long by default, and that is exactly what fills the KV cache.

The load generator streams these and measures the two standard serving metrics per request. TTFT, time to first token, is how long until the first token arrives. TPOT, time per output token, is how fast tokens stream after that.

the captured run · overload.json 
$ uv run loadgen.py --requests 64 --concurrency 32 --max-tokens 1024 --json-out capture/overload-requests.json
requests: 64 ok, 0 failed, wall 43.4s
throughput: 1439.2 output tok/s, 1.47 req/s
ttft  p50 147ms  p90 1863ms  p99 5344ms
tpot  p50 16.4ms  p90 19.1ms  mean 17.1ms

Note 64 ok, 0 failed. Overloading a serving stack does not produce errors, it produces latency. The server admits what fits, queues the rest, and every request eventually completes. The cost shows up in TTFT, which is 147ms at the median but 5.3 seconds at p99, while TPOT stays flat at 17ms. The timeline shows why. Drag across any chart.

KV cache used, % 0 50 100 requests running waiting 0 20 40 preemptions since start 0 8 16 output tokens, tok/s 0 1.2k 2.4k GPU busy, % compute (SM) memory bus 0 50 100 0 10 20 30 40s GPU power, W 0 150 300 0 10 20 30 40s
drag across a chart to inspect
t+0s KV – running – waiting – preempted – tok/s – SM – power –

One overload run, captured from Prometheus at 1s resolution. Drag to inspect any moment, or press replay.

The KV cache fills, and once it is full the scheduler starts evicting running requests to make room, which is the preemptions panel. An evicted request keeps its place but has to recompute its cache when it gets back in. Requests that cannot start at all sit in the waiting queue, and that queue time is exactly the exploding TTFT. A request at the back waits seconds for its first token, then streams at the same 17ms per token as everyone else. Under overload, TTFT is queueing, not compute.

The GPU panels show the other half. A request has two phases. Prefill processes the whole prompt in one parallel pass, then decode generates tokens one at a time. This run is almost all decode, and every generated token reads essentially all 8 GB of weights once, so the memory bus saturates before anything else. The GPU’s compute units, the SMs (streaming multiprocessors) in the chart, sit at 100% while the memory bus is above 90%. That is this card doing all it can.

A single request decodes at about 87 tok/s. The batch sustains 1439 tok/s. That gap is continuous batching: while one request waits on memory for its next token, the same weight reads serve the other 31.

Tracing one conversation with tool calls

The overload run showed the whole herd under pressure. This is the opposite experiment, one conversation on an idle server with a single trace following it end to end. The same queue, prefill and decode from above reappear inside every vLLM span, now for one request you can name.

To make the trace worth reading I gave the model a job it cannot do alone, solving xx=10x^x = 10 to ten decimal places. Thinking alone, a 4B model burns thousands of tokens on longhand Newton iteration for four correct decimals at best. With a python tool it writes the method, runs it, and answers exactly.

The tool is model-written code in a throwaway container with no network:

agent.py 
proc = subprocess.run(
    ["podman", "run", "--rm", "-i", "--network=none", "--memory=256m",
     "--pids-limit=64", "docker.io/library/python:3.12-alpine", "python", "-"],
    input=code.encode(), capture_output=True, timeout=60,
)

The agent loop opens a root span, every model turn and tool execution becomes a child, and vLLM joins the same trace through the W3C traceparent header. One session, one trace.

0s 2s 4s 6s 8s 10s 12s GPU busy 100% 0 GPU memory 16GiB 0 GPU power 250W 0 agent.solve 13.23s llm.call 3.08s llm_request 3.07s tool.python 286ms llm.call 4.43s llm_request 4.43s tool.python 283ms llm.call 2.34s llm_request 2.34s tool.python 287ms llm.call 2.49s llm_request 2.49s
agent vllm prefill part click a span for its attributes

The whole agent session as one distributed trace, GPU telemetry from the same capture aligned on top. Click any span. Model turns carry the thinking and the decision, tool spans carry the exact code and its stdout, vLLM spans break down into queue, prefill and decode.

Click through and the session tells its own story. The model reaches for numpy, the sandbox answers ModuleNotFoundError, the next thinking block switches to the standard library, and the last span carries the answer. The GPU works during every model turn and idles during every tool call.

The answer, x=ln⁡10W(ln⁡10)≈2.5061841456x = \frac{\ln 10}{W(\ln 10)} \approx 2.5061841456 with WW the Lambert W function, came out exact to all ten places in three tool calls and 1030 completion tokens. The longhand attempts spent 14k tokens for four decimals.

What the gen_ai spans actually contain

The per-request latency split that powers the phase bars above is the most useful thing vLLM’s tracing emits, and it is not in the OpenTelemetry GenAI semantic conventions at all. The conventions moved to their own repository and are still marked Development, so this is a comparison against a moving target. Checked as of July 2026, on vLLM 0.24.0, the current release, and the attribute names are unchanged on vLLM main. What vLLM emits versus what the spec says:

vLLM emitsthe spec says
span name llm_request{operation} {model}, so chat unsloth/qwen3.5-4b
—gen_ai.operation.name, gen_ai.provider.name, the only two required attributes
gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokensrenamed to gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
—gen_ai.usage.reasoning.output_tokens
gen_ai.request.id holding the completion idthat is gen_ai.response.id
gen_ai.latency.time_in_queue, time_in_model_prefill, time_in_model_decode—
—gen_ai.response.time_to_first_chunk, measured from the client

This is not a stale corner of vLLM either. The community keeps adding to the nonstandard set rather than renaming it toward the spec.

The empty cells tell the story. vLLM’s queue, prefill and decode timings have no equivalent in the spec, and the spec’s only timing attribute is measured from the client. The conventions treat the model as a black box over the wire, so the view from inside the server has no standard home. The empty cell in the reasoning row points the opposite direction. The spec can report thinking tokens separately, vLLM never fills that in, and this model spends most of its output tokens on thinking.

The mismatch matters because backends key off the standard names. Datadog’s LLM Observability ingests OTel GenAI spans natively and maps gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.provider.name into its own schema for latency, tokens and cost. Dynatrace’s AI Observability does the same over OTLP, alongside OpenLLMetry and OpenInference. Point either at vLLM’s spans as they are and none of that special handling should fire, because the span name is wrong, the required attributes are absent, and the token counts hide under the old names. The traces still arrive as plain spans, but the LLM-specific views have nothing to grab. Until vLLM catches up with the spec, the practical fix is renaming attributes in an OTel collector transform between vLLM and the backend.

I work at Dynatrace. This is a personal post written on my own time and hardware, not reviewed or endorsed by my employer, and both vendor descriptions above come from public documentation.

Footnotes

  1. FlashInfer JIT-compiles its CUDA kernels with nvcc on first startup, and CUDA 13.2 refuses Fedora 44’s gcc 16 as a host compiler. Fedora packages gcc15 for exactly this case, and NVCC_PREPEND_FLAGS='-ccbin /usr/bin/g++-15' points every nvcc invocation at it. The first startup spends ten minutes compiling, later ones reuse the cache. ↩

  2. Qwen3.5-4B is a vision-language model. At startup vLLM profiles worst-case memory including the vision encoder at default batch limits, and that profiling pass alone OOMs a 16 GB card even though the weights are only 8 GB. --limit-mm-per-prompt '{"image": 0, "video": 0}' skips the vision path entirely. ↩

  3. GNOME and Chrome hold one to two fluctuating gigabytes of VRAM that vLLM’s --gpu-memory-utilization accounting cannot see. At 0.8 the engine OOMed while allocating the KV cache, at 0.7 everything fits with the KV cache taking what is left, about 33k tokens. ↩