QTM-1008 - Hydrating Crow's Nest Monitoring Targets (Compiling Prometheus scrape configs from manifests)

Living Document Notice
Published 2026-09-17. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Hydrating Crow’s Nest Monitoring Targets (Compiling Prometheus scrape configs from manifests)

Summary

Dynamic service discovery systems such as Consul or etcd introduce cluster consensus overhead and background network chatter. For bounded fleets, static file-based discovery allows Prometheus to monitor nodes with zero external service dependencies. Quartermaster compiles node manifests directly into Prometheus file discovery targets.

The Cost of Dynamic Service Discovery Daemons

Monitoring infrastructure often relies on dynamic service discovery backends like Consul, Zookeeper, or Kubernetes DNS. These systems introduce operational complexity. Running service discovery agents on small edge nodes consumes valuable system resources and generates continuous heartbeat network traffic.

If a consensus leader election stalls or network latency spikes across nodes, Prometheus drops scrape targets, producing false alert storms in the operations center. For small to medium fleets where node additions and decommissionings occur through planned administrative actions, dynamic discovery represents unnecessary overhead.

Crow’s Nest, the centralized observability stack for Bosun infrastructure, uses Prometheus file-based service discovery (file_sd_configs). Quartermaster translates static node manifests into clean JSON target manifests.

Target Compilation Architecture

The Quartermaster compiler processes the declared hardware and service manifests from /etc/quartermaster/nodes/*.yaml and emits normalized Prometheus targets to /etc/crowsnest/targets/nodes.json.

Whenever a new node or service registers in version control, the Quartermaster CI runner regenerates the targets file and syncs it to the Crow’s Nest server. Prometheus automatically detects file changes using inotify without requiring daemon restarts.

Exporter Target Port and Metric Allocations

Every monitored node exposes a defined set of lightweight exporters. The compiler assigns scrape endpoints deterministically based on manifest service declarations.

Exporter Service	Listen Port	Scrape Interval	Target Metric Domain	Egress Protection
node_exporter	9100	15 seconds	CPU, RAM, disk I/O, network statistics	Restricted to monitoring IP via nftables
caddy_metrics	2019	30 seconds	HTTP request rates, TLS handshake latencies	Bound strictly to loopback interface
directus_probe	8055	60 seconds	SQLite pool health, CMS response codes	Requires internal bearer token authentication
smart_exporter	9633	120 seconds	NVMe percentage used, disk wear telemetry	Runs on a low-frequency scrape timer

Generated Prometheus Target Specification

The output file emitted by the compilation pipeline matches the standard Prometheus file_sd format.

[
  {
    "targets": [
      "10.100.0.12:9100",
      "10.100.0.12:9633"
    ],
    "labels": {
      "node": "edge-lon-01",
      "tier": "edge-proxy",
      "chassis": "1u-rack",
      "datacenter": "lon-facility-3",
      "env": "production"
    }
  },
  {
    "targets": [
      "10.100.0.13:9100",
      "10.100.0.13:8055"
    ],
    "labels": {
      "node": "core-lon-01",
      "tier": "cms-backend",
      "chassis": "blade",
      "datacenter": "lon-facility-3",
      "env": "production"
    }
  }
]

Compilation Pipeline Implementation

The build pipeline uses a lightweight script to transform YAML node declarations into the target JSON structure.

#!/bin/sh
set -eu
 
TARGET_FILE="/etc/crowsnest/targets/nodes.json"
NODES_DIR="/etc/quartermaster/nodes"
 
mkdir -p "$(dirname "$TARGET_FILE")"
 
# Temporary buffer for valid JSON assembly
TMP_TARGETS=$(mktemp)
echo "[" > "$TMP_TARGETS"
 
FIRST=true
for MANIFEST in "$NODES_DIR"/*.yaml; do
  [ -e "$MANIFEST" ] || continue
  
  NODE_ID=$(awk -F': ' '/identifier:/ {print $2}' "$MANIFEST" | tr -d ' "')
  MGMT_IP=$(awk -F': ' '/management_ip:/ {print $2}' "$MANIFEST" | tr -d ' "')
  TIER=$(awk -F': ' '/tier:/ {print $2}' "$MANIFEST" | tr -d ' "')
  
  if [ "$FIRST" = true ]; then
    FIRST=false
  else
    echo "," >> "$TMP_TARGETS"
  fi
 
  cat <<EOF >> "$TMP_TARGETS"
  {
    "targets": ["${MGMT_IP}:9100"],
    "labels": {
      "node": "${NODE_ID}",
      "tier": "${TIER:-standard}"
    }
  }
EOF
done
 
echo "]" >> "$TMP_TARGETS"
mv "$TMP_TARGETS" "$TARGET_FILE"
chmod 0644 "$TARGET_FILE"
echo "Compiled targets written to $TARGET_FILE"

Verification and Reload Invocations

Prometheus monitors target files via file watchers, eliminating service interruptions:

# Manually validate target file syntax
promtool check config /etc/prometheus/prometheus.yml
 
# Verify target discovery status through Prometheus API
curl -s http://localhost:9090/api/v1/targets | grep -o '"discoveredLabels":{[^}]*}'

---

• Directus Target: quartermaster
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [QTM-1008 - Hydrating Crow’s Nest Monitoring Targets (Compiling Prometheus scrape configs from manifests)](QTM-1008 - Hydrating Crow’s Nest Monitoring Targets (Compiling Prometheus scrape configs from manifests))