HRHalden Ross ~/$
Book a call
booking new engagements — from August 2026mainUS Mountain (UTC−7) · overlaps US + EU morningsresp < 1 business day
IoT / data pipelines · 2023

A streaming pipeline that ingests 2B events/day

Device telemetry was overwhelming a batch warehouse. I built a streaming ingestion and aggregation pipeline that kept dashboards live at fleet scale.

Platform engineer (contract)Fixed-scope project9 weeksScale & Reliability →
measured outcome
2B/day
events ingested
<1s
fleet dashboard queries
−61%
pipeline cost per event
99.97%
ingestion uptime

// The problem

Two billion daily device events were loaded by hourly batch, so fleet dashboards were always behind and costs were ballooning as volume grew.

// The approach

Stream-first ingestion with cheap cold storage.
  1. Designed a streaming ingestion path with Kafka partitioned by device cohort.
  2. Built stream processors for windowed aggregation with exactly-once semantics.
  3. Stored rollups in ClickHouse for sub-second fleet queries and raw events in cheap object storage.
  4. Added schema registry and backpressure so a device firmware bug couldn’t take the pipeline down.
  5. Cut cost with tiered retention and right-sized compute.
stack
GoKafkaClickHouseS3KubernetesTerraformPrometheus
Start a conversation Next: Cut checkout p99 latency 85% before a 9× launch

Case studies, metrics, prior companies, and testimonials in this template are illustrative and anonymized for demonstration; they do not name or represent any real client or employer and are not a guarantee of future results. Outcomes vary by system, team, and constraints.

next step

Have something that has to ship — or stay up?

Book a 30-minute call. We’ll talk through what you’re building, what hurts, and whether I’m the right person for it. No charge, no pitch — and a scoped plan before any work begins.

Book a call hello@haldenross.example