A Flash Sale Is a Load Test You Didn't Schedule

Published 2015-04-27 · Updated 2026-06-23 · By Alan Wizemann

Topics: Engineering & Development, Product Strategy & Development

When the Lilly Pulitzer collection launched at Target, the site buckled. It was a well-publicized mess, and I am not going to pretend it was anything else. What I can do, which is more useful than an apology, is tell you what actually broke, because the popular explanation (too much traffic) is the least helpful version of the story. The collection was a limited run of a beloved brand at accessible prices, which is to say it was engineered, quite deliberately, to create exactly the frenzy it created, and demand at launch came in at several times our previous all-time peak in orders per hour. That much, on its own, was not the problem; we knew it would be big. The problem was everything the spike exposed underneath it, in parts of the system we had been quietly tolerating for years.

The first crack appeared before the launch even started. The product pages had been published early enough that the search engines had already indexed them, which meant a determined guest could find a live page and buy items hours ahead of the official drop. The most popular pieces began selling out before we had sold anything on purpose. That created confusion, call-center volume, and a trail across social media, all before the event we had planned for had begun. The lesson is one that is genuinely easy to forget: your launch does not begin when you announce it, it begins the moment anything at all is reachable.

The real failure came at the scheduled launch, and it was architectural rather than a matter of raw volume. We had a database that sat behind every experience at once, so the desktop site, the mobile web, and the apps all leaned on the same shared spine. When the apps took the brunt of the demand, the strain did not stay contained to the apps. It propagated straight through that shared database and began degrading everything else along with it – new guests could not start sessions, the people who got bounced retried through a different channel, and each retry piled more load onto the spine that was already failing. A problem that started in one place became a problem everywhere, precisely because everything was connected at the bottom in a way nobody had been forced to think about until that afternoon.

Here is the detail I think about most. We could throttle the web traffic to relieve pressure, but we could not throttle the app traffic anywhere near as quickly, because we had never built the controls to do it. So the only lever available, the only way to save the underlying system, was the bluntest one imaginable: cut the apps' access to the backend entirely and take them fully down to keep the rest standing. When your only two options are "fully up" and "fully off," you do not really have control. You have a switch. Recovery came eventually, once the database locks cleared and we brought traffic back, but by then most of the collection was gone, sold during the chaos itself, and that produced its own second wave of frustrated guests. There is no clean ending to an incident like this; you restore the system and then spend days absorbing the human aftermath of it.

So what do you actually take from a day like that? The first thing is that a hyped drop is a load test you never got to schedule, where every assumption in your architecture gets audited at once, on your most visible day, by real customers rather than a test harness. The shared dependencies you have been tolerating precisely because they have never been a problem will choose that exact moment to become the problem. If you have experiences that are supposed to be independent but quietly share one fragile spine, you do not have independent experiences at all – you have a single experience wearing several costumes. The second thing is that throttling is a feature you build on purpose, not an afterthought you reach for in the moment. The ability to gracefully limit each channel on its own, letting some traffic through while protecting the core, is the whole difference between a slow day and a full outage, and we had it in one place and not another. That one gap defined how the entire incident played out.

The third thing, and the one most people skip, is that the experience layer can soften what the infrastructure cannot. A great deal of the pain that day was not the outage itself, but guests pushing all the way into a purchase flow only to discover the thing they came for was already gone. Surfacing that reality earlier, before someone has emotionally committed to a checkout, turns a furious customer into a merely disappointed one. That is not an infrastructure fix; it is a product decision, and it mattered as much as anything we did on the backend. I want to be careful not to tie too tidy a bow around this, because the honest version is that we got caught. The demand was foreseeable, the weak point was knowable, we had performance-tested to a number, and the real world walked straight past that number without slowing down. That is the uncomfortable part of scale: the system behaves beautifully right up to the edge of what you tested, and tells you almost nothing about what waits one step beyond it. What I took from it was not a fear of big moments but a sharper set of questions going into the next one. Not "can our servers handle the traffic," which only invites you to throw capacity at the problem and feel safe, but what is actually shared underneath these experiences that should not be, and where can we throttle as opposed to where are we stuck with a switch, and what will a guest see at the precise moment the thing they came for runs out. You cannot out-provision a bad assumption. The traffic just turns the lights on long enough for everyone to finally see it.