Administration¶

This guide covers operational tasks for running HyperbyteDB in production: monitoring, backup/restore, compaction tuning, cluster management, and the debug CLI.

Monitoring¶

Prometheus Metrics¶

HyperbyteDB exposes a Prometheus-compatible metrics endpoint at GET /metrics on the same port as the API (default 8086). There is no separate metrics port.

Key metrics:

Cluster-specific metrics:

Prometheus scrape configuration¶

scrape_configs:
  - job_name: 'hyperbytedb'
    static_configs:
      - targets: ['hyperbytedb:8086']
    metrics_path: /metrics
    scrape_interval: 15s

For clusters, scrape each node individually.

Logging¶

Logs are written to stderr. Control verbosity with the [logging] config section:

Set format = "json" for structured output compatible with log aggregation tools (Loki, Elasticsearch, etc.).

Health endpoint¶

GET /health returns:

{"status": "pass", "message": "ready for queries and writes"}

Always returns 200 as long as the HTTP server is running. In cluster mode, a node in Draining or Leaving state still responds to /health but rejects writes.

Backup and Restore¶

Create a backup¶

hyperbytedb backup --output /backups/hyperbytedb-$(date +%Y%m%d)

The backup directory contains:

Backups can run while HyperbyteDB is serving traffic. RocksDB checkpoints are consistent point-in-time snapshots.

Restore¶

# 1. Stop HyperbyteDB
# 2. Restore (overwrites configured directories)
hyperbytedb restore --input /backups/hyperbytedb-20240115
# 3. Start HyperbyteDB
hyperbytedb serve

Restore overwrites the configured data_dir, wal_dir, and meta_dir.

Compaction Tuning¶

Compaction merges small Parquet files into larger ones, improving query performance and reducing file count.

Key parameters¶

Trigger on-demand compaction¶

curl -sS -XPOST 'http://localhost:8086/internal/compact'

This runs compact_all(), which compacts every measurement regardless of min_files_to_compact.

Monitor compaction¶

Watch hyperbytedb_parquet_files_count per measurement. After compaction, file counts should decrease. If they keep growing, check: - hyperbytedb_compaction_errors_total for failures - Logs for compaction error details - Whether min_files_to_compact is set too high

Cluster Operations¶

Debug CLI¶

The hyperbytedb-debug binary queries a running cluster over HTTP for on-call inspection (membership, replication lag, manifests, compaction). Source: src/bin/hyperbytedb_debug.rs. Build: cargo build --release --bin hyperbytedb-debug.

Global options:

--nodes may be set with HYPERBYTEDB_NODES.

hyperbytedb-debug \
  -n "10.0.0.1:8086,10.0.0.2:8086,10.0.0.3:8086" \
  --scheme http --timeout 10 \
  status

Subcommands:

manifest: --node <addr> chooses the node (default: first in --nodes).

metrics: --filter <substring> (default hyperbytedb_cluster).

compact: --node <addr> compacts only that node; --all runs against every address in --nodes; --compact-timeout (default 300 seconds) caps per-node wait.

hyperbytedb-debug -n "127.0.0.1:8086" compact
hyperbytedb-debug -n "a:8086,b:8086,c:8086" compact --node b:8086
hyperbytedb-debug -n "a:8086,b:8086,c:8086" compact --all

Graceful drain¶

To remove a node from the cluster without data loss:

curl -sS -XPOST 'http://node-to-remove:8086/internal/drain'

The drain procedure: 1. Sets node state to Draining (rejects new writes with 503). 2. Flushes all WAL entries to Parquet. 3. Waits for replication acks from all peers (up to 60 seconds). 4. Verifies data consistency via Merkle tree comparison. 5. Notifies peers of departure. 6. Sets state to Leaving.

Self-repair¶

In cluster mode, the compaction service runs automatic Parquet self-repair: - Verified compaction hashes cold data buckets against authoring peers and replaces divergent files. - Membership-driven repair discovers missing data slices from peer manifests.

These run on the compaction interval. Tune with verified_compaction_age_secs, self_repair_enabled, and max_repair_checks_per_cycle in the [compaction] config section. See Configuration for details.

Background Services¶

HyperbyteDB runs several background services as Tokio tasks:

All services shut down gracefully on ctrl+c: the flush service performs a final flush, then all service handles are awaited.

See Also¶

• Configuration — Full reference for all tuning parameters
• Troubleshooting — Diagnosing common issues
• Common workflows — Backup procedures, monitoring setup