Administration¶

Monitoring, backup and restore, retention, cluster operations, and background services.

Monitoring¶

Prometheus Metrics¶

HyperbyteDB exposes a Prometheus-compatible metrics endpoint at GET /metrics on the same port as the API (default 8086). There is no separate metrics port.

Key metrics:

Cluster-specific metrics:

Prometheus scrape configuration¶

scrape_configs:
  - job_name: 'hyperbytedb'
    static_configs:
      - targets: ['hyperbytedb:8086']
    metrics_path: /metrics
    scrape_interval: 15s

For clusters, scrape each node individually.

Logging¶

Logs are written to stderr. Control verbosity with the [logging] config section:

Set format = "json" for structured output compatible with log aggregation (Loki, Elasticsearch, and similar).

Environment variable equivalents:

HYPERBYTEDB__LOGGING__LEVEL=info
HYPERBYTEDB__LOGGING__FORMAT=json
HYPERBYTEDB__LOGGING__DETAILED_TRACE=true

Distributed tracing¶

HyperbyteDB exports OpenTelemetry traces over OTLP HTTP when logging.otlp_endpoint is set. This is independent of detailed_trace: you can export sampled traces in production without enabling per-phase span creation on every request.

Traces are tagged with service.name=hyperbytedb (override with OTEL_SERVICE_NAME).

The root docker-compose.yml ships Alloy, Loki, Tempo, and Grafana with trace-to-log correlation preconfigured. After starting the stack:

1. Open Grafana → Explore → Tempo.
2. Search for service.name=hyperbytedb.
3. Run a few writes and queries, then inspect span timings.

See Configuration for all logging keys.

Statement summary¶

When statement_summary.enabled = true, recently executed TimeseriesQL statements are available at GET /api/v1/statements. Each entry includes the normalized query text, digest, execution time, and error status. Useful for correlating slow queries with Tempo traces and Loki logs.

Health endpoint¶

GET /health returns:

{"status": "pass", "message": "ready for queries and writes"}

Always returns 200 as long as the HTTP server is running. In cluster mode, a node in Draining or Leaving state still responds to /health but rejects writes.

Backup and Restore¶

Create a backup¶

hyperbytedb backup --output /backups/hyperbytedb-$(date +%Y%m%d)

The backup directory contains:

Backups can run while HyperbyteDB is serving traffic. RocksDB checkpoints are consistent point-in-time snapshots. For off-node copies, use your operator backup CRD or object storage tooling.

Restore¶

# 1. Stop HyperbyteDB
# 2. Restore (overwrites configured directories)
hyperbytedb restore --input /backups/hyperbytedb-20240115
# 3. Start HyperbyteDB
hyperbytedb serve

Restore overwrites the configured wal_dir, meta_dir, and chDB session data directory.

Retention¶

Retention policies are enforced by a background loop that runs ALTER TABLE … DELETE against expired rows in each measurement's MergeTree table. Tune frequency with [retention].interval in config. See Configuration.

Cluster Operations¶

Cluster inspection (HTTP)¶

Use the built-in HTTP endpoints for on-call inspection:

curl -s http://node1:8086/cluster/metrics | jq .
curl -s http://node2:8086/internal/sync/manifest | jq .

Compare manifests across nodes to spot replication lag or catalog drift.

Graceful drain¶

To remove a node from the cluster without data loss:

curl -sS -XPOST 'http://node-to-remove:8086/internal/drain'

The drain procedure: 1. Sets node state to Draining (rejects new writes with 503). 2. Flushes all WAL entries into chDB MergeTree tables. 3. Waits for replication acks from all peers (up to 60 seconds). 4. Notifies peers of departure. 5. Sets state to Leaving.

Cluster sync¶

In cluster mode, the Raft leader periodically compares /internal/sync/manifest responses from peers and may mark peers as needing sync. New nodes pull metadata and WAL deltas via the sync APIs. See Deep Dive: Clustering.

Background Services¶

HyperbyteDB runs several background services as Tokio tasks:

All services shut down gracefully on ctrl+c: the flush service performs a final flush, then all service handles are awaited.

See Also¶

• Configuration — Full reference for all tuning parameters
• Troubleshooting — Diagnosing common issues
• Common workflows — Backup procedures, monitoring setup