The Million-Tenant Problem: Why Your OpenSearch Service Architecture Breaks at Scale

Almost every search application is multi-tenant, whether or not anyone called it that. A store has product categories. A SaaS product has customers. A logging platform has teams, an e-commerce site has sellers, a B2B app has accounts. Each of those is a tenant: a slice of data, and the queries that go with it, that has to stay separate from every other slice. If you run Amazon OpenSearch Service, odds are you are already in the tenancy business.

And here is the thing about tenancy: it is almost free when you have a few tenants, and it quietly turns into the whole problem when you have a lot of them. A design that is clean and obvious at a hundred tenants can fall apart at a hundred thousand, for reasons that have nothing to do with the code being wrong. Take a concrete case. You are running a fintech app, peer-to-peer payments or a neobank, and millions of users each expect to open their phone, pull up six months of history, and see it instantly. That data behaves like logs, streaming in and aging out, but it has to perform like search, because a person is staring at a spinner. It lives in the gap between the two, and the conventional wisdom for either end does not fit.

I have watched teams wrestle with this exact problem. They stand up a textbook Amazon OpenSearch Service domain, follow every log-analytics best practice, demo it, and it screams. Beautiful on the happy path, right up until a certain scale. Then the queries slow down, the costs balloon, and the architecture that purred along at 10,000 tenants starts making the noise hardware makes right before the magic smoke escapes. The question I hear next is always the same: “We thought we were doing this right. What changed?” Usually nothing changed. The design followed the playbook faithfully, right down to copying the shard sizes from a blog post. The playbook just was not written for this workload, and scale is where that bill comes due. Let me walk through where the conventional wisdom runs out, and then the handful of moves that actually hold up at a million tenants. Where the conventional wisdom runs out

The first trap is the pure-logging approach. The standard playbook says ingest fast, roll over indices daily, keep three days hot, then tier the rest off to cheaper storage. Shard size? Go big, 50 GB is fine, because you are optimizing for write throughput. This is excellent advice for application logs that nobody reads unless something is on fire.

It falls apart the moment you serve individual tenant queries at scale. Every query fans out across every shard in the index. A terabyte in 50 shards means one user tapping their phone becomes a single request fanning out to 50 servers, all to answer somebody checking last month’s coffee purchases. The cost of coordinating that scatter-gather swamps the benefit of parallelism, especially when each tenant’s data is a thin slice smeared across all 50 shards. The cluster was tuned for throughput, and now it is being asked for the one thing t…