Observability Stack — Prometheus, Grafana, Loki, Mimir

Overview

The Grafana observability stack provides a unified pipeline for metrics (Prometheus/Mimir), logs (Loki), and visualization (Grafana). This reference covers each component's role, common deployment patterns, and the architectural tradeoffs between monolithic and microservice deployments.

Component roles

Component	Purpose	Port
Prometheus	Metrics collection, short-term storage, alerting	9090
Grafana	Dashboards, visualization, alerting UI	3000
Loki	Log aggregation, storage, querying	3100
Promtail / Alloy	Log collection agent (ships logs to Loki)	9080
Mimir	Long-term, horizontally scalable metrics storage	9009
Alertmanager	Alert routing, grouping, silencing	9093

Prometheus

Architecture

Prometheus scrapes metrics from instrumented targets via HTTP, stores them in a local time-series database, and evaluates alerting rules. For high availability, run two identical instances scraping the same targets — use an HA pair, not a cluster.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100", "node-2:9100"]

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

rule_files:
  - "alerts/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Alerting rules

# alerts/node.yml
groups:
  - name: node
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for 10 minutes (current: {{ $value }}%)"

      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"

PromQL quick reference

# Rate (per-second average over time window)
rate(http_requests_total[5m])

# Increase (total increase over time window)
increase(http_requests_total[1h])

# Histogram quantile
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Aggregation
sum(rate(http_requests_total[5m])) by (status_code)
avg(node_cpu_seconds_total) by (instance)

# Top K
topk(5, rate(http_requests_total[5m]))

# Arithmetic
rate(http_errors_total[5m]) / rate(http_requests_total[5m]) * 100

Loki

Loki indexes log metadata (labels) but not the log content itself, keeping storage efficient.

LogQL quick reference

# Basic query
{app="myapp"} |= "error"

# Filter with regex
{app="myapp"} |~ "ERROR|FATAL"

# Exclude pattern
{app="myapp"} != "DEBUG"

# Parse JSON logs and filter
{app="myapp"} | json | status_code >= 500

# Aggregate
sum(count_over_time({app="myapp"} |= "error" [5m]))

# Rate of errors over time
rate({app="myapp"} |= "error" [5m])

Promtail configuration

# promtail-config.yaml
server:
  http_listen_port: 9080

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log

  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container

Mimir

Mimir is a horizontally scalable, highly available, long-term storage backend for Prometheus metrics. It accepts Prometheus remote write and serves PromQL queries.

When to use Mimir

Need	Prometheus	Mimir
Short-term metrics (hours/days)	✓	✓
Long-term retention (months/years)	✗	✓
High availability (multi-node)	HA pair only	✓ (clustered)
Multi-tenancy	✗	✓
Global view (multi-cluster)	Federation only	✓

Key architecture components

Component	Purpose
Distributor	Validates and distributes incoming samples to ingesters
Ingester	Buffers recent data in memory, flushes to object storage
Querier	Handles PromQL queries across ingesters and store-gateways
Store-gateway	Serves long-term data from object storage (S3, GCS)
Compactor	Merges and deduplicates blocks in object storage
Ruler	Evaluates recording and alerting rules

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for generating, collecting, and exporting telemetry data. It unifies traces, metrics, and logs into a single framework, replacing vendor-specific agents with a vendor-neutral pipeline.

Architecture

┌───────────────────────────────────────────────────────┐
│  Application                                           │
│  ┌─────────┐  ┌──────────┐  ┌────────────┐            │
│  │  Traces  │  │  Metrics │  │    Logs     │            │
│  │  (SDK)   │  │  (SDK)   │  │  (SDK/Bridge)│           │
│  └────┬─────┘  └────┬─────┘  └─────┬──────┘            │
│       └──────────────┼─────────────┘                   │
│                      ▼                                 │
│            ┌─────────────────┐                         │
│            │ OTel Collector  │  (agent/sidecar/gateway)│
│            │  - Receivers    │                         │
│            │  - Processors   │                         │
│            │  - Exporters    │                         │
│            └───────┬─────────┘                         │
└────────────────────┼───────────────────────────────────┘
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
    ┌─────────┐ ┌───────┐ ┌─────────┐
    │  Tempo  │ │ Mimir │ │  Loki   │
    │ (traces)│ │(metrics)│ │ (logs) │
    └─────────┘ └───────┘ └─────────┘

Collector configuration

The collector runs as a DaemonSet (agent) or Deployment (gateway) and routes telemetry to backends:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite/mimir:
    endpoint: http://mimir:9009/api/v1/push
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite/mimir]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Auto-instrumentation (Node.js)

# No code changes — just require the module at startup
node --require @opentelemetry/auto-instrumentations-node app.js

// tracing.js — minimal manual setup
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4317" }),
});
sdk.start();

Tempo — Distributed Tracing

Tempo is Grafana's distributed tracing backend. Unlike Jaeger or Zipkin, Tempo is designed to be cost-effective at scale by separating search from storage — it only indexes trace IDs and service names, keeping trace data in cheap object storage.

When to use Tempo

Need	Tempo	Jaeger
Massive trace volumes (billions/day)	✓ (S3-backed)	Limited by Cassandra/ES
Full-text search on spans	✗ (use Loki for logs)	✓ (Elasticsearch)
Tight Grafana integration (TraceQL, Exemplars)	✓	Partial
Simple single-binary deployment	✓ (monolithic mode)	✓

TraceQL quick reference

# Find traces where a service errored
{ .http.status_code >= 500 }

# Find slow traces
{ duration > 3s }

# Trace with specific attribute
{ resource.service.name = "checkout" && .http.method = "POST" }

# Aggregate: count of spans by service
{ } | count() by resource.service.name

# Histogram of span durations
{ } | histogram(duration) by name

Tempo deployment modes

Mode	Description	When
Monolithic	Single binary, local storage	Dev, small-scale (< 10M spans/day)
Scalable monolithic	Single binary, S3 backend	Medium scale, simpler than microservices
Microservices	Separate scaling per component	Large scale (> 100M spans/day)

# Start Tempo in monolithic mode with S3 backend
docker run -d --name tempo \
  -p 3200:3200 \
  -v ./tempo.yaml:/etc/tempo.yaml \
  grafana/tempo:latest -config.file=/etc/tempo.yaml

Tempo + Loki + Grafana integration

Grafana auto-correlates traces with logs when both are configured as data sources. Add a traceID to your structured logs and Grafana shows a "Tempo" button next to log lines — click it to see the full trace for that request.

Grafana Alloy

Alloy is Grafana's unified agent — the successor to Promtail, Grafana Agent, and the OpenTelemetry Collector. It collects logs, metrics, and traces using a single binary and pipeline-based configuration language.

Replacing Promtail with Alloy

# alloy-config.alloy — replaces promtail-config.yaml
logging {
  level = "info"
}

loki.source.file varlogs {
  targets = [
    { __path__ = "/var/log/*.log", job = "varlogs" },
    { __path__ = "/var/log/nginx/*.log", job = "nginx" },
  ]
  forward_to = [loki.write.default.receiver]
}

loki.source.docker containers {
  host = "unix:///var/run/docker.sock"
  labels = {
    job = "docker",
  }
  forward_to = [loki.write.default.receiver]
}

loki.write default {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Alloy + OpenTelemetry (metrics + traces)

// otelcol.receiver.otlp receives OTLP data from instrumented apps
otelcol.receiver.otlp default {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    metrics = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch default {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
    traces  = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.prometheus default {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write mimir {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

otelcol.exporter.otlp tempo {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

End-to-end deployment (docker-compose)

A complete observability stack in a single docker-compose.yml — suitable for development and small production environments:

# docker-compose.yml — full LGTM stack (Loki, Grafana, Tempo, Mimir) + Alloy
version: "3.8"

services:
  # ── Metrics ──────────────────────────────────────────
  mimir:
    image: grafana/mimir:latest
    command: ["-ingester.ring.replication-factor=1", "-target=all", "-config.file=/etc/mimir.yaml"]
    ports: ["9009:9009"]
    volumes:
      - ./mimir.yaml:/etc/mimir.yaml
      - mimir-data:/data

  # ── Logs ─────────────────────────────────────────────
  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]
    command: -config.file=/etc/loki/local-config.yaml

  # ── Traces ───────────────────────────────────────────
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    ports: ["3200:3200", "4317:4317"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo

  # ── Agent (logs + traces collection) ─────────────────
  alloy:
    image: grafana/alloy:latest
    ports: ["12345:12345"]
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy
      - /var/log:/var/log:ro
      - /var/run/docker.sock:/var/run/docker.sock

  # ── Visualization ────────────────────────────────────
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_INSTALL_PLUGINS: grafana-lokiexplore-app
    volumes:
      - ./datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
      - grafana-data:/var/lib/grafana

volumes:
  mimir-data:
  tempo-data:
  grafana-data:

# Start the full stack
docker compose up -d

# Verify all components
curl http://localhost:3000       # Grafana UI
curl http://localhost:3100/ready # Loki
curl http://localhost:3200/ready # Tempo
curl http://localhost:9009/ready # Mimir

Architecture tradeoffs

Monolithic deployment

Best for: Small teams, single cluster, simple operations.

Pros	Cons
Simple to deploy and debug	No high availability (single points of failure)
Low resource overhead	No horizontal scaling
Single config file	Cannot isolate noisy neighbors

Deploy: Single Prometheus, Loki, Grafana instance per cluster. Use docker-compose or a single Helm chart.

Microservice / HA deployment

Best for: Multi-cluster, multi-tenant, production at scale.

Pros	Cons
Horizontal scaling per component	Complex deployment and networking
Fault isolation (component failure doesn't take down everything)	Higher resource overhead
Multi-tenancy with isolation	Requires object storage (S3/GCS)
Long-term retention via Mimir	Operational complexity

Deploy: Mimir in microservice mode, Loki in scalable mode (separate read/write paths), Grafana with PostgreSQL backend.

Recommended production pattern

┌─────────────────────────────────────────────────┐
│  Kubernetes Cluster                              │
│                                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │ Promtail │  │  Node    │  │   App    │       │
│  │(DaemonSet)│ │ Exporter │  │  /metrics│       │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘       │
│       │             │             │              │
│       ▼             ▼             ▼              │
│  ┌─────────┐  ┌────────────────────────┐        │
│  │  Loki   │  │  Prometheus (HA pair)  │        │
│  │(scalable)│ │  → remote_write → Mimir │       │
│  └────┬────┘  └────────────────────────┘        │
│       │                                         │
└───────┼─────────────────────────────────────────┘
        │
        ▼
┌───────────────┐     ┌──────────────┐
│  Grafana      │────▶│  Mimir (S3)  │
│  Dashboards   │     │  Long-term   │
│  Alerts       │     │  metrics     │
└───────────────┘     └──────────────┘

Grafana

Data source setup

# datasource.yml (provisioning)
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

  - name: Loki
    type: loki
    url: http://loki:3100
    access: proxy

  - name: Mimir
    type: prometheus
    url: http://mimir-nginx:9009/prometheus
    access: proxy

Dashboard JSON model essentials

{
  "panels": [
    {
      "title": "Request Rate",
      "targets": [
        {
          "expr": "rate(http_requests_total[5m])",
          "legendFormat": "{{status_code}}"
        }
      ],
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
    }
  ],
  "templating": {
    "list": [
      {
        "name": "instance",
        "type": "query",
        "query": "label_values(node_cpu_seconds_total, instance)"
      }
    ]
  }
}

Alertmanager

# alertmanager.yml
route:
  group_by: ["alertname", "severity"]
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 4h
  receiver: "slack-critical"
  routes:
    - match:
        severity: critical
      receiver: "pagerduty"
    - match:
        severity: warning
      receiver: "slack-warnings"

receivers:
  - name: "slack-critical"
    slack_configs:
      - channel: "#alerts-critical"
        api_url: "https://hooks.slack.com/services/..."
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

  - name: "pagerduty"
    pagerduty_configs:
      - routing_key: "<pagerduty-key>"
        severity: critical

  - name: "slack-warnings"
    slack_configs:
      - channel: "#alerts-warnings"
        api_url: "https://hooks.slack.com/services/..."

See also

• OpenTelemetry — instrumentation and collector configuration
• SRE Architecture — deployment patterns and SLOs
• GitOps with ArgoCD — continuous deployment patterns
• AWS VPC Architecture Overview — network architecture