Case study 2 of 6
DeadReckon
Ships hiding from AIS, found in a public day of Gulf of Mexico traffic replayed through ScyllaDB, Redis and Kafka.
- stack
- Python
- ScyllaDB
- Redis
- Kafka (Redpanda)
- H3
- PyTorch
- ONNX
- FastAPI
- deck.gl
- Docker
- Prometheus
384
rendezvous alerts in three replayed hours, with per-cell weights
tick = 2,485 with every cell weighted 1.0; 85% fewer, same hours, same vessels
0.146 km
next 5 min position error, 14,273 test windows
tick = dead reckoning, 0.189 km; at 60 min physics wins, 4.698 against 4.714
F1 0.41
vessel type from motion alone, 337 unseen vessels, six classes
tick = majority class, 0.091; accuracy 0.423 against 0.376
Where a silent ship should be, against where it came back
Pick a vessel from the fixture, choose the fix where its transponder goes dark and how long it stays dark. The faint paths are every other vessel in the same four hours. The instrument projects the last course and speed forward, draws the radius the projection is trusted to (2 nm, or 30% of the distance run), and measures how far off the real reappearance landed. A ratio at or above 1 is a gap_reappear alert in the repo, with severity ratio over 4.
6,246 positions of 168 vessels, 06:00 to 10:00 UTC, one fix per vessel per 300 s; eight moving tracks shown. Projection and radius from deadreckon/geo.py, the verdict from processing/gaps.py, on a flat km plane around 28.85 N, 94.6 W, refitted to each reading.
A tanker on a straight run reappears inside the radius after an hour and the gap closes with no alert. A tug working the approaches reappears kilometres off its last heading, and the same rule flags it. The detector only opens a gap for a vessel that was moving, away from any port zone, and silent for 45 minutes.
Three ways a ship hides, and why a plain silence detector fails
Ships report their position over AIS every few seconds. A ship that wants to be unseen turns the transponder off, meets another ship at sea, or declares a type it does not move like. Coastal receivers cannot tell "switched off" from "out of range", so a detector that fires on every silence fires on every cell no antenna covers.
DeadReckon stores every fix and judges each silence against how quiet that H3 cell is at that hour of the week, learned from the days before. A cell where most vessels sit still at anchor also stops counting two neighbours as a meeting. A cell with too little history keeps weight 1.0, so an unknown area trusts the vessel over the model. The data is a live aisstream.io feed or a public NOAA day replayed at 60× to 600×.
The replayed day is a Monday and the history is Friday to Sunday, so no weekday row matches and every lookup falls through to the hour pooled over all days. That fallback alone cuts rendezvous alerts by 85%. The gap_open count moving from 161 to 169 is sweep boundaries meeting Kafka batch boundaries differently on each 600× run, not the model. A replay is a demo; the pin is the offline run below.
The transformer (three causal blocks, 311k parameters, 643 s of CPU training) is the best next-step predictor, 23% under dead reckoning, and that next step is what the anomaly score uses. On 30 and 60 minute rollouts the learned errors compound and constant velocity has the better median at every horizon, so the corridor the API serves stays physics. Train and validation are split by vessel hash; the test day is one weekday after three training days, and the unseen rows are vessels absent from those days.
Kinematics alone separate passenger and pleasure craft reasonably. Cargo is mostly called tanker (same speed band, same lanes) and tugs scatter everywhere. Labels are self-declared AIS types and every labelled vessel seen on a training day is dropped from the test, so the F1 is on vessels the classifier never saw. The map treats a type mismatch as a weak signal; the badge needs 0.7 confidence and a 0.9 mismatch.
| measured | result |
|---|---|
| 384k positions drained into Scylla and Redis, one writer | 182 s end to end, 2,094-2,496 positions/s |
| the same replay, two writers | 90 s end to end, 2,809-3,224/s each |
| ScyllaDB CPU during the single-writer run | 8%; the Python writer (pydantic validation and JSON) is the bottleneck, not the store |
| Redis rebuilt from Scylla after FLUSHDB | 4,233 vessels, 0 rows differing |
| one of three Scylla nodes stopped mid-replay | the writer kept 3,400 to 3,600 positions/s at LOCAL_QUORUM |
| poison messages | to ais.dlq with the reason; the consumer keeps its offset and nothing stops |
ScyllaDB is the append-only event store, one table per question and no joins; the same fix is written by vessel and by cell, raw tables expire after 90 days and an hourly roll-up is kept. Redis holds the last four hours as a derived view with a TTL on every entry. Kafka messages are keyed by MMSI, offsets commit after the sinks, and the detector's clock is wall time live and event time in replay.
The offline run that pins the detector
The fixture above runs through the real writer and the real detector on fakeredis, with no broker or database, in about ten seconds. It raises 17 alerts (14 gap_open, 1 gap_reappear, 2 rendezvous) with flat weights and 16 with a coverage table that marks one cell as an anchorage. CI diffs both summaries against committed baselines, so moved detection logic shows up as one failing field, not as an impression. The suite is 608 tests at 100% line and branch coverage, and the pure modules passed mutation testing.
What it cannot do
- Terrestrial AIS cannot separate "off" from "out of range"; the weights need weeks of history, and the demo has three days.
- Rendezvous trusts the declared navigation status; a ship parked in a moored fleet is invisible to it.
- The type classifier sits at F1 0.41, so a mismatch is a hint for a reviewer, not evidence of spoofing.
- Four days of one region; seasonal and long-horizon generalisation is untested, and the models were still improving at the last epoch of a 40 minute CPU budget.
- Detector state lives in one process. The map needs the internet for deck.gl and tiles.