Resource Sizing Guide¶

Guidelines for provisioning CPU, memory, disk, and cluster topology for HyperbyteDB workloads.

What sizing means¶

HyperbyteDB sizing has three mostly independent dimensions:

Write rate alone is a poor way to pick CPU and RAM. The ingest path — HTTP accept, parse, WAL append, and periodic flush to chDB — is optimized for high throughput. On SSD or NVMe storage, a single node can sustain around 1 million points/sec end-to-end (write through to queryable storage) with modest CPU usage. That does not mean you need 16 cores because you ingest 500K points/sec.

Query load is the main reason to add CPU and RAM. Dashboards, alerts, and ad-hoc TimeseriesQL that scan wide time ranges, aggregate with GROUP BY, or run concurrently will consume far more CPU and memory than the write path at the same ingest rate.

Plan each dimension separately, then add headroom for your heaviest expected query load.

Query load — primary CPU and RAM driver¶

Size CPU and RAM from how many queries run at once and how heavy they are — not from points/sec ingest.

What makes a query heavy:

• No time filter — scans all retained data for a measurement.
• Wide time range — e.g. 30 days of raw data in one chart.
• Aggregations — MEAN, COUNT, GROUP BY time(1h), GROUP BY host over large spans.
• Concurrency — many Grafana panels, alert rules, or API clients querying at once.

Practical suggestions:

• Add time bounds to dashboards and alerts (WHERE time > now() - 24h or similar).
• Cap concurrent queries so heavy scans do not pile up:

  [server]
  max_concurrent_queries = 16
  
  [chdb]
  pool_size = 4
  

  Set max_concurrent_queries ≥ pool_size so flush and queries can overlap. See Configuration.
• If queries are slow but CPU is idle, you may be I/O-bound on disk — check storage type and free space before adding cores.
• If CPU is saturated while ingest stays healthy, reduce concurrency or simplify queries before scaling write throughput assumptions.

There is no distributed query fan-out in a cluster: each node serves queries from its own chDB. Balance read traffic across nodes (client-side or via hyperbytedb-proxy). See V1 Stable Scope.

Write throughput — disk and flush, not CPU¶

High ingest rates mainly affect disk IOPS, data growth, and whether the flush pipeline keeps up — not how many CPU cores you need.

What high write rates do affect:

• Daily data volume — feeds the storage budget below.
• WAL disk use — short-lived; typically 1–5% of total data volume.
• Flush lag — if ingest outruns flush, the WAL grows until catch-up; this shows up in metrics before CPU spikes.

What high write rates usually do not require:

• Proportional CPU scaling — the WAL uses bounded in-memory structures and group commit.
• Proportional RAM scaling — RocksDB WAL footprint is capped by design; Arrow WAL cache size follows flush batch settings, not ingest rate linearly.

If writes are slow or the WAL grows without bound, check disk IOPS and flush settings first, not core count.

Storage — retention and data volume¶

Disk sizing follows how much data you keep, not query concurrency.

Estimating capacity¶

1. Estimate raw daily volume:

   daily_raw_bytes ≈ points/sec × 86,400 × average_point_size
   

   A typical line-protocol point is ~100 bytes (measurement, two tags, several fields).

2. Apply compression — numeric time-series often compresses 5–10× on disk (LZ4 default).

3. Multiply by retention days and add 20–30% free space for MergeTree compaction.

Example: 100K points/sec, ~100 bytes/point, 30-day retention: - Raw ≈ 864 GB/day → on-disk ≈ 90–170 GB/day after compression → ~3–5 TB for 30 days plus headroom.

Adjust up for longer retention, larger points, or low-cardinality tag explosion.

WAL (RocksDB)¶

chDB MergeTree (data)¶

Compaction headroom¶

MergeTree compaction runs in the background. Leave 20–30% free disk space so compaction can proceed without blocking ingest.

Memory¶

Rule of thumb: Pick RAM from the query profile table above. A node ingesting 500K points/sec with two dashboards needs far less RAM than a node ingesting 50K points/sec with fifty concurrent heavy aggregates.

Cluster sizing¶

In a multi-node cluster, each node ingests, replicates, and queries its own chDB copy. Scale cluster size for read capacity, fault tolerance, and ingest fan-out — not by multiplying CPU per point/sec.

Example starting points (per node, before replication overhead):

Tuning by symptom¶

See Troubleshooting for step-by-step fixes.

After deployment — what to watch¶

Once the node is running with representative ingest and query traffic, use Prometheus metrics (see Administration) to confirm sizing:

Start with defaults, deploy with realistic query patterns (same dashboards and alerts you expect in production), then adjust concurrency limits and hardware based on what you observe.

See Also¶

• Configuration — Tuning parameters (max_concurrent_queries, pool_size, flush settings)
• Administration — Metrics and monitoring
• Troubleshooting — Query timeouts, memory, and flush issues
• V1 Stable Scope — Supported topologies and availability model