Posted by Mike Sandlas
Filed in Business 37 views
There is a moment in every platform's life when you stop worrying about whether people will show up and start worrying about whether your infrastructure can survive them actually showing up. For us, that moment came on a Tuesday afternoon in March when a single high-profile political event sent our trading volume through the roof and our engineering team into a controlled panic. Within forty minutes, we had gone from a comfortable 400 trades per second to nearly 6,000. By the end of the week, after some frantic architectural decisions and a few sleepless nights, we had rebuilt the backbone of our system to comfortably handle 10,000 trades per second and beyond.
This is the story of how we did it, what we learned, and what we wish we had built from the beginning.
If you have ever built an e-commerce site or a standard financial app, you might think trading infrastructure is just a faster version of the same problem. It is not. Prediction markets are fundamentally different from both traditional exchanges and consumer applications because every single trade depends on real-time probability recalculation, order book matching, and event resolution logic all happening simultaneously. You cannot queue a trade and process it later. The price that existed 200 milliseconds ago may be completely irrelevant by the time your response reaches the user.
When we first launched our prediction marketplace platform development project, we architected it the way most early-stage startups do. We had a monolithic backend, a relational database doing heavy lifting, and a WebSocket layer for real-time updates. It worked beautifully at low volume. It started showing cracks around 300 concurrent active traders. By the time we hit a genuine traffic spike, we were genuinely close to melting down.
The core problem was that our system was designed around requests, not around events. Each trade was a synchronous round trip. The database was the source of truth and the bottleneck. Our order matching engine lived inside our API layer, which meant it competed for resources with everything else the API was doing. We had built something logical and reasonable, and it was completely wrong for the use case.
The first thing we did, even before we touched infrastructure, was separate concerns. Our order matching engine had to become its own service, isolated from everything else, running on its own resources with its own optimized data structures. This single decision gave us a 4x throughput improvement before we changed a single line of database code.
A matching engine for a prediction market is not just a FIFO queue. It has to handle multiple outcome legs, implied probability calculations, liquidity aggregation across different contract types, and settlement triggers that fire when external data sources confirm an event outcome. Building this as a standalone service written in Go, with an in-memory order book and a write-ahead log for durability, was the decision that made everything else possible.
Once the matching engine was isolated, we could scale it independently, profile it independently, and optimize it without touching the rest of the platform. We went from matching roughly 600 trades per second to over 4,000 on the same hardware just by removing the I/O contention with our API layer.
Here is the part of the story that most engineering blogs skip because it is embarrassing. We held on to our PostgreSQL setup for too long because we liked it, understood it, and had already built a lot of reporting infrastructure on top of it. That was a mistake that cost us six weeks of performance problems we kept trying to patch around rather than fix.
The honest answer to what it takes to scale a prediction market is this: your primary trade execution path cannot touch a traditional relational database in the hot path. Full stop. You need an in-memory data layer for active order books and open positions, with asynchronous persistence happening behind the scenes. We moved to a Redis-backed order book with Kafka as our event bus and PostgreSQL demoted to a reporting and settlement database that received writes asynchronously. Trade confirmation latency dropped from an average of 120 milliseconds to under 8 milliseconds.
This is also one of the most important things to understand when teams start thinking about the cost to build prediction market platform infrastructure at scale. The compute costs are real, but they are not the biggest variable. The biggest variable is the engineering time required to properly design the data layer. Getting this right early versus retrofitting it later is often the difference between a six-month rebuild and a six-week feature sprint.
Once your architecture is event-driven and your matching engine is isolated, horizontal scaling becomes tractable. But prediction markets have a unique constraint that most scaling guides do not address: you cannot simply spin up ten matching engine instances and load balance across them without partitioning your markets correctly. Two trades on the same contract cannot be matched by two different engine instances without a coordination mechanism, and coordination mechanisms kill throughput.
Our solution was market-level sharding. Each prediction market contract is assigned to exactly one matching engine shard based on a consistent hash of the market ID. Trades are routed to the correct shard at the API gateway level. New shards can be added and markets rebalanced during low-volume windows. This gave us near-linear horizontal scalability for the first time.
We also completely separated our read and write paths. Price discovery, market depth, and recent trade history are served from a read replica layer that receives updates via our Kafka event stream. Users querying current market odds never touch the write path at all. This took an enormous amount of read pressure off our core systems and allowed us to serve thousands of concurrent users browsing open markets without any measurable impact on trade execution performance.
One of the things that shaped our thinking significantly during this rebuild was studying how regulated prediction market operators had approached the same problem at scale. Kalshi's prediction market software development practices became something of a reference point for us, not because we were building the same regulated product, but because they had navigated the specific challenge of combining financial-grade reliability requirements with the real-time throughput modern users expect.
What struck us most was their emphasis on correctness first, speed second. In prediction markets, an incorrect trade settlement is not just a bug, it is a financial dispute. Every optimization we made had to be validated against a correctness guarantee. We built an extensive simulation harness that replayed historical trade sequences against any new matching engine build before we would let it touch production traffic. This added time to our deployment cycle, but it also meant we never had a settlement error in production during our entire scaling effort. That record mattered enormously for user trust.
We also borrowed the concept of circuit breakers aggressively. If any component of our execution path started responding outside its latency budget, we would automatically stop accepting new orders for affected markets and notify users, rather than allowing degraded but incorrect execution. Users will forgive downtime. They will not forgive being told a trade executed at one price when it actually cleared at another.
At our current scale, the stack that handles production traffic looks like this. Our API gateway runs on a cluster of Go services behind a Layer 4 load balancer, handling authentication, rate limiting, and market routing. The matching engine cluster runs on dedicated bare metal nodes rather than cloud VMs because the latency variance from cloud hypervisor scheduling was measurable and unacceptable for our tightest SLA requirements. Kafka handles all inter-service communication and event sourcing. Redis Cluster manages our in-memory order books and position state. PostgreSQL handles settlement, reporting, and historical data. A dedicated event resolution service monitors our oracle integrations and triggers settlement workflows when market conditions are met.
Our observability stack is arguably as important as any of the above. We track p50, p95, and p99 latencies for every stage of the trade execution pipeline in real time. We alert on p99 degradation before it ever becomes visible to users. We run continuous synthetic trade load against production to detect latency regressions before real user traffic exposes them.
On peak days we now comfortably clear 10,000 to 12,000 trades per second with p99 execution latency under 15 milliseconds. Our matching engine cluster uses about 60 percent of capacity on those peaks, which gives us meaningful headroom before we need to scale further.
If we were starting this over, the three things we would do differently are these. First, we would build the matching engine as a separate service from day one rather than treating it as a future optimization. Second, we would design our data layer around an event-sourced, in-memory-primary model from the first trade, even if early volume does not demand it, because retrofitting this later is genuinely painful. Third, we would invest in the simulation and correctness testing harness before we wrote a single line of matching logic, because in prediction markets, being fast and wrong is worse than being slow and right.
Scaling a prediction market platform is not just an infrastructure challenge. It is a product challenge, a correctness challenge, and an organizational challenge. The teams that get it right are the ones who treat all three with equal seriousness.
The 10,000 trades per second number is satisfying to say out loud. But the number we are actually most proud of is zero, which is the number of incorrect settlement events we have had since we rebuilt this system properly. In a market where trust is the product, that number matters more than throughput.
Building at this scale is hard, and it is also genuinely one of the most interesting engineering problems in consumer fintech right now. If you are earlier in your journey and trying to figure out where to start, the architecture decisions described here are the ones that move the needle. Get the fundamentals right first, and the throughput follows.
