Deep Dive: Read Path¶

This document traces the HyperbyteDB query path from HTTP request to InfluxDB-compatible JSON response. It covers TimeseriesQL parsing, ClickHouse SQL translation against native MergeTree tables, chDB execution, tombstone filtering, and result formatting.

Table of Contents¶

1. Overview
2. HTTP Query Handler
3. TimeseriesQL Parser
4. Query Dispatch
5. SELECT Execution
6. Tombstone Injection
7. chDB Execution
8. Result Formatting
9. Cluster Query Behavior
10. Metrics

1. Overview¶

Client GET/POST /query
       |
       v
+------------------------------------+
| HTTP Handler (query.rs)            |
|  - Extract q, db, epoch, params    |
|  - Substitute bind parameters      |
|  - Delegate to QueryService        |
+------------------------------------+
       |
       v
+------------------------------------+
| QueryServiceImpl::execute_query()  |
|  - Parse TimeseriesQL              |
|  - Timeout wrapper                 |
|  - Dispatch each statement         |
+------------------------------------+
       |
       v (SELECT statements)
+------------------------------------+
| execute_measurement_query()        |
|  - Resolve native table name       |
|  - Translate AST → ClickHouse SQL  |
|  - Inject tombstone predicates     |
|  - Execute via chDB                |
|  - Parse JSONEachRow → Series      |
+------------------------------------+
       |
       v
+------------------------------------+
| chDB (embedded ClickHouse)         |
|  SELECT FROM MergeTree tables      |
|  Returns JSONEachRow               |
+------------------------------------+

Key source files: adapters/http/query.rs, application/query_service.rs, timeseriesql/, adapters/chdb/query_adapter.rs.

2. HTTP Query Handler¶

File: src/adapters/http/query.rs

Entry points¶

• GET /query?q=...&db=...
• POST /query with form-encoded or query-string parameters

POST merges form body parameters into query string parameters (body takes precedence).

Parameters¶

Timeout¶

The handler wraps execution in tokio::time::timeout using server.query_timeout_secs.

3. TimeseriesQL Parser¶

Module: src/timeseriesql/ (Influx-compatible query language)

The parser is a hand-rolled recursive descent parser in parser.rs. It splits multi-statement input on ; and dispatches on the first keyword:

SELECT parsing builds a SelectStatement AST with fields, FROM clause, WHERE, GROUP BY (including time(5m) buckets), ORDER BY, LIMIT/OFFSET, SLIMIT/SOFFSET, fill options, and subqueries.

Expression parsing uses precedence climbing for OR, AND, comparisons, arithmetic, and function calls.

4. Query Dispatch¶

File: src/application/query_service.rs

execute_query() parses the input into Vec<Statement> and dispatches each:

SHOW and DDL statements never touch chDB tables directly — they read or mutate the RocksDB metadata store (and Raft log in cluster mode).

5. SELECT Execution¶

Function: execute_measurement_query()

Steps¶

1. Resolve measurement — Extract name from FROM clause; handle regex measurements by querying metadata for matches and UNION ALL across tables.
2. Retention policy — Default RP from metadata when omitted.
3. Native table name — Build the physical table identifier via domain/chdb_naming (same naming as the write path).
4. Translate — to_clickhouse::translate_native_table(stmt, &table, column_mapping) converts the AST to ClickHouse SQL targeting the MergeTree table directly (not file() over external files).
5. Tombstones — Append AND NOT (predicate) for each stored tombstone (see §6).
6. Execute — QueryPort::execute_sql() runs the SQL through chDB.
7. Parse results — Transform JSONEachRow into InfluxDB v1 series format.

Translation highlights¶

The internal alias __time avoids collision with the raw time column and is renamed back to time in the result parser.

6. Tombstone Injection¶

DELETE statements do not immediately remove rows from MergeTree tables. Instead:

1. The WHERE predicate is converted to a ClickHouse SQL fragment.
2. A tombstone record is stored in metadata: tombstone:{db}:{measurement}:{uuid}.
3. In cluster mode, the DELETE mutation is replicated via Raft.

On SELECT, inject_tombstone_predicates() loads all tombstones for the measurement and appends exclusion predicates to the WHERE clause. Deleted data is hidden at query time without rewriting stored rows.

7. chDB Execution¶

Adapter: ChdbQueryAdapter (adapters/chdb/query_adapter.rs)
Port: QueryPort

Session model¶

chDB runs inside spawn_blocking. HyperbyteDB opens separate query and write connection pools to the same session_data_path (chdb.query_pool_size and chdb.write_pool_size, default 4 each). The read path uses the query pool only; each connection has its own ChdbClient mutex, so concurrent query tasks overlap when query_pool_size > 1. Ingest/flush uses the write pool, so heavy queries do not block inserts. Tune server.max_concurrent_queries (≥ query_pool_size) to cap in-flight query tasks.

Output format¶

Queries use FORMAT JSONEachRow. Each line is a JSON object:

{"__time":"2024-01-15 10:00:00","host":"server01","mean_usage_idle":42.5}

8. Result Formatting¶

The query service transforms JSONEachRow into InfluxDB v1 series format:

1. Parse each line as a JSON object.
2. Rename __time back to time.
3. Convert ClickHouse datetime strings to nanosecond timestamps.
4. Apply the epoch parameter (ns/us/ms/s integers or RFC3339 strings).
5. Group rows by tag combination into separate SeriesResult objects.
6. Apply SLIMIT/SOFFSET for series-level pagination.

SELECT INTO¶

When the query includes an INTO clause, results are written back through the ingestion path as new points in the target measurement.

9. Cluster Query Behavior¶

Full-copy cluster (default)¶

When [sharding] enabled = false, each node executes queries against its local chDB tables. There is no distributed scatter-gather for data queries — clients should route to a healthy peer or use a load balancer.

Schema mutations (CREATE DATABASE, DELETE tombstones, CQ definitions) go through Raft for ordering and are replicated to all peers.

Sharded cluster (experimental)¶

When [sharding] enabled = true, SELECT and SHOW statements on sharded measurements are scatter-gather:

1. Resolve the measurement's shard map and select overlapping regions.
2. Translate TimeseriesQL to ClickHouse SQL and inject per-region series_id predicates.
3. For each region, execute locally (if this node is an Active peer) or POST to /internal/shard/query / /internal/shard/metadata on Active region peers.
4. Merge partial results on the coordinator into InfluxDB-compatible JSON.

The scatter loop tries Active peers in order (self → primary → replicas), respects scatter_max_peer_attempts and scatter_peer_timeout_ms, and fails over to replicas when the primary is unreachable. Stale shard epochs abort without retry.

For replication, sync, and sharding control-plane details, see Deep Dive: Clustering.

10. Metrics¶

When sharding is enabled, additional metrics track region selection and scatter behavior — see Deep Dive: Clustering.

When enabled, StatementSummary records normalized query text and timing for GET /api/v1/statements.

Related documents¶

• Architecture — hexagonal overview
• Write path — how data reaches MergeTree tables
• API & TimeseriesQL Reference — HTTP endpoints and syntax