For years InfinityLog stored logs the obvious way — a relational database, a table per day, a row per event. It worked, until it didn't. Here's why we tore out the storage layer and rebuilt it on a cutting-edge storage technology, and what changed for the ISPs running it.
The problem with a fat row per event
The old approach keeps every field of every event together on disk, wrapped in per-row overhead and indexes. For a handful of records that's fine. For an ISP writing tens of millions of flows a day it's brutal: disks fill in weeks, and a query for one subscriber has to walk enormous per-day tables that were never built for fast scans. Operators told us the same thing — the logs were there, but searching them was something you avoided.
A smarter way to store an event
Our cutting-edge storage technology groups similar values together and compresses them, instead of paying full price for a bloated row every time. The same event that once cost well over a hundred bytes now lands in roughly 16 bytes on disk. A network that used to need a 16 TB HDD comfortably fits on a 2 TB SSD — about 8× smaller — and because a search only reads exactly what it needs, a full day of traffic comes back in under a second. We don't publish the internals, and deliberately so: the efficiency is the product, and it's a hard thing for a competitor to copy.
Migrating without losing a single line
The catch: you can't take a compliance log server offline to re-platform it, and you can't lose history mid-cutover. So the migration is a dual-write: new events flow into the new store and the legacy store at the same time, reads are served from whichever holds the range you ask for, and only once the new store is proven do we retire the old write path. Ingestion never stops. No day goes dark.
What the operator actually feels
- Storage bills drop — months of retention on a small SSD instead of a rack of spinning disks.
- Search becomes something you use — investigations that were an afternoon are now a query.
- Nothing changes at the router — the same syslog stream, the same fields, the same compliance record.
None of this shows up on a spec sheet the day you sign. It shows up six months later, the first time a lawful request lands with a deadline and you answer it before the call ends.