Building SkyStream — Sai Rajesh Tanikonda

Why I Built This

Flight data is one of the best publicly available real-time datasets. Aircraft broadcast their position, altitude, speed, and heading every few seconds via ADS-B transponders — and APIs like airplanes.live aggregate this into a free global feed of roughly 9,000 aircraft at any given moment.

I wanted to build something that demonstrated a complete data engineering stack. Not a tutorial toy, but a system with real throughput, real latency constraints, and real architectural tradeoffs. Flight tracking hit all of those boxes.

System Architecture

The system has two paths: a hot path for live positions and a cold path for historical storage. Everything runs in Docker containers orchestrated with Docker Compose on a Linux VPS.

airplanes.live API
       │
       ▼
   Producer (Python)
       │  Kafka topic: raw-flights
       ▼
   Spark Structured Streaming
       │
       ├──▶ Redis          ← live state cache (hot path)
       └──▶ TimescaleDB    ← historical trail storage (cold path)
                │
                ▼
         FastAPI WebSocket API
                │
                ▼
         React + deck.gl frontend

The Stack

Layer	Technology
Data source	airplanes.live ADS-B API
Message broker	Apache Kafka (4 partitions, 24hr retention)
Stream processor	Spark Structured Streaming (5s micro-batches)
Time-series DB	TimescaleDB (PostgreSQL extension)
Live cache	Redis (HSET per aircraft)
Backend API	FastAPI + WebSockets
Frontend	React + deck.gl + MapLibre GL
Deployment	Docker Compose on Linux VPS

The Producer & Kafka

A Python service polls airplanes.live every 10 seconds, pulling ~9,000 aircraft states per cycle. Each state is validated with Pydantic and published to the raw-flights Kafka topic. The geographic coverage is configurable via a bounding box environment variable — the contiguous US requires a ~1,700nm radius query. Each cycle publishes around 8,800 messages in under one second.

Kafka acts as the buffer between the producer and Spark, providing three key guarantees:

Spark can fall behind without data loss (24-hour log retention)
Multiple consumers can read the same topic independently
The producer never blocks on downstream slowness

Spark Structured Streaming

Spark reads from the Kafka topic, parses the JSON payloads, enriches each record with a computed flight_phase, and writes to two sinks every 5 seconds.

vertical_rate > +2 m/s   →  CLIMBING
vertical_rate < -2 m/s   →  DESCENDING
on_ground = true          →  GROUND
otherwise                 →  CRUISE

Sink 1 — TimescaleDB: stores every position update as a time-series row. TimescaleDB's hypertable partitions data automatically by time, making range queries fast.

Sink 2 — Redis: stores the latest state per aircraft in a hash (HSET flights {icao24} {json}). The WebSocket server reads from here for live positions with minimal latency.

A critical decision: Redis writes happen before Postgres writes, and Postgres failures never block the Redis write. Independent sinks, independent failure modes.

The Map Interface

The map is built with deck.gl on top of MapLibre GL. deck.gl's IconLayer renders thousands of aircraft icons with GPU acceleration — smooth at 9,000+ points without breaking a sweat. Aircraft icons rotate to match their heading and are colored by flight phase (green climbing, blue cruise, orange descending, gray ground). Trail lines use PathLayer showing each aircraft's recent path as a colored polyline.

The UI is fully responsive. On mobile, the aircraft info panel becomes a bottom sheet and the tracked flights panel becomes a horizontal scroll strip at the top.

Problems I Actually Hit

⚠ Zero aircraft after redeploy

Spark replayed old Kafka messages → all Redis entries had stale timestamps → WebSocket filtered everything → empty map.

Fix

Delete the Spark checkpoint and restart from the latest Kafka offset.

⚠ Postgres failure taking down the live map

Spark's process_batch wrote to Postgres first, then Redis. A missing database raised an exception that skipped the Redis write entirely.

Fix

Write to Redis first. Wrap the Postgres write in its own try/except.

⚠ Data source 404 on global radius

adsb.fi returns 404 for any radius over ~3,000nm.

Fix

Switched to airplanes.live which supports up to a 10,000nm global radius.

⚠ Map auto-following locked aircraft

The map re-centered every 5 seconds even when the user tried to pan away.

Fix

Detect isDragging in deck.gl's onViewStateChange and clear the tracking lock on user interaction.

What I Learned

Checkpoints are double-edged. Spark checkpoints give fault tolerance but cause stale data replays after downtime. Always have a plan for resetting them.
Design sinks to be independent. One sink failing should never take down another.
Coordinate TTLs and staleness filters. If your consumer filters records older than 2 minutes, your producer must write fresher than that under all failure scenarios.
deck.gl handles scale beautifully. WebGL-accelerated rendering handles 9,000+ moving points without issue. The layer API is clean and composable.
WebSocket fan-out at scale needs care. Sending 9,000 aircraft to 100 concurrent users every 5 seconds is ~90MB/s. Viewport filtering is the next optimization.

What's Next

Viewport-aware broadcasting — only send aircraft visible in the user's current map view
Aircraft type enrichment from a static ICAO database
Anomaly detection — flag unusual squawk codes (7500 hijack, 7700 emergency)
Historical playback — scrub through a time range and replay recorded flight positions

Live Demo →GitHub →