Metric Stream

Metric Stream is a purpose-built telemetry ingestion pipeline designed for environments where sensor data arrives faster than a traditional time-series database can absorb it.

Architecture

Producers write events to Redis Streams using XADD. A pool of Go consumer goroutines reads with XREADGROUP, processes and enriches the events (unit conversion, anomaly flagging, threshold evaluation), and forwards the result to the downstream store.

func (w *Worker) Run(ctx context.Context) error {
    for {
        entries, err := w.redis.XReadGroup(ctx, &redis.XReadGroupArgs{
            Group:    w.group,
            Consumer: w.id,
            Streams:  []string{w.stream, ">"},
            Count:    100,
            Block:    time.Second,
        })
        if err != nil { continue }
        for _, e := range entries[0].Messages {
            w.process(ctx, e)
        }
    }
}

Throughput

In benchmarks on 4 consumer instances, the pipeline sustains 52k events/second with P99 end-to-end latency of 18ms. The Redis consumer group ensures exactly-once delivery semantics even if a worker crashes mid-batch.

Status

Deployed in a staging environment with production rollout planned for Q3 2026. Pending load testing at 100k events/second target.