Real-time collaborative grid
Live collaboration for many readers and a few editors, built from simple pieces.
This one was a short timeline, a lot of constraints, and a lot of little details. Real-time collaboration is hard even when you have every tool you want. We didn't. We wanted it cheap and simple, and we didn't want to build some big distributed system for a feature we weren't even sure people would use.
So we took something really hard and broke it into simple pieces. Some pieces we just chose not to solve. Honestly, that's enterprise software at its finest.
What we didn't build
We didn't do merging or conflict resolution. Cells lock while someone's editing them, so conflicts just don't happen. I know how the merging version works. It wasn't worth the complexity for us.
We didn't use WebSockets. Most people in the grid are reading, not editing. One-way streams are cheap to send to a lot of readers, and I didn't want to scale two-way connections for people who are mostly just watching.
And nobody gets events they shouldn't. You only subscribe to the data you're allowed to see. Nobody gets every event and filters it on their side.
Most of this is pretty obvious once you lay it out. Anybody could get there. But there was one part I thought was really clever.
The watermark
When you open the grid, you need two things: a snapshot of the data and the live stream of changes. The stream opens first. Then the snapshot loads, and that takes time. While it's loading, events keep coming in.
We originally built it without a watermark, and we got all sorts of weird bugs. The problem is the snapshot doesn't tell you what you missed. Some of the events you buffered are already in the snapshot and some aren't, and you can't tell which. Apply an old event on top of newer data and your grid is wrong, and you don't even know it. We didn't have replay, so we couldn't just rewind and catch up.
I was looking at this like, how do I fix it quickly?
Everything in our system goes through Redis, even inside a single service. The write path publishes to Redis and the stream reads from Redis, so every event passes through one place. That means Redis can stamp every event with a counter that only goes up. That's the watermark.
The watermark only lives in Redis. I didn't touch the database or change the data shape. When a snapshot is taken, it makes one call to Redis to grab the current watermark first. The client compares what it buffered against that number, drops anything at or below it, and applies the rest. Sometimes an event applies twice, and that's fine, because applying the same edit twice gives you the same result. One extra call. That's essentially free.
#42 ≤ 43 · dropped
When Redis resets
You might be thinking: what happens if Redis restarts? Doesn't the counter go back to zero? Yes, and I ran into exactly that.
If the service knows Redis reset, it's easy: tell the client to grab a fresh snapshot. But if the service doesn't know, I needed a fallback. So if the counter doesn't exist, it starts again from the current time in microseconds, plus one. The counter only goes up by one per event, which is way slower than a clock counting microseconds. So a restart always lands ahead of anywhere the counter has been. The watermark doesn't go backwards.
The fully airtight version would need to remember the last watermark somewhere. For how unlikely that failure is, it wasn't worth it.
I found out later this is close to something called a hybrid logical clock. At the time it was just the simplest way I could think of to make sure the number always goes up.
What I owned
The backend. Real-time transport, locks, pub/sub, the database side, and the shape of how clients sync. A teammate built the interface, which is really rare for me, but this time I was needed underneath.
How it turned out
We shipped it fast and people use it. The locks aren't perfect. But the system stayed simple enough that I always knew how it would behave, and on a timeline like this, that's most of the job.