During the 25 million concurrent user test on June 11, P95 latency spiked from 37ms to 1,815ms at Phase 2 onset. That number looks alarming. It generated questions from everyone who saw the data.
It was not a scaling failure. It was an early indicator that triggered a corrective action. Understanding why matters more than the number itself.
What the Timeline Actually Showed
- Phase 2 load begins. user-catalog RPS surges from 7,400 to 72,000 in under 2 minutes.
- P95 spikes to 1,815ms. user-catalog has reached its configured pod ceiling at 720 RPS per pod.
- KEDA triggers ingress gateway scale-out. This is 13 minutes after the spike started.
- Conclusion: the spike was caused by pod saturation on a single non-critical service, not by a scaling lag across the platform.
The ingress gateway scale-out helped manage subsequent load on other services. But it could not address user-catalog directly. That service cannot scale beyond its configured maxReplicas. Raising the ceiling fixes the spike entirely.
Why P95 Is Not the Right Signal to Watch
Not all latency metrics carry equal operational weight during a live event. Here is a practical framework for interpreting what you are seeing:
P50 is the primary health indicator.
If P50 is stable, the majority of viewers are having a good experience. In the June 11 test, P50 held at 4ms from baseline through peak burst. That is the signal that mattered most. It never moved.
P95 is the early warning signal.
P95 elevation indicates a saturating service tier before it fails completely. The 1,815ms spike was visible 13 minutes before any KEDA event fired. It is a leading indicator, not an outage signal. When P95 rises without a corresponding P50 movement, the problem is isolated.
P99 is the tail experience.
Driven by the slowest 1% of requests. Often a specific slow database query, a cold cache, or a pod restart. Important for SLA tracking but not the primary operational signal during a live event.
DC flags are client-side resets.
313 RPS of DC flags at 458,000 total RPS equals 0.1%. These are viewers whose devices reset the connection, typically network jitter or mobile context switches. Not server errors.
The Value of Architectural Isolation
The reason the P95 spike stayed isolated is architectural. The platform separates workloads into three tiers: the critical path covering authentication, DRM, and stream authorization; operational tools like Istio and Karpenter; and non-critical services like personalization and recommendations.
user-catalog is a non-critical service. When it saturated, it degraded gracefully. Viewers received the stream with sensible defaults rather than a failure page. The critical path services, device-token, playback-auth, heartbeat, sustained sub-5ms response times throughout.
This means the platform's load ceiling is defined by the critical path, not by the weakest supporting service. Without that isolation, a user-catalog ceiling would have been a platform outage.
What Good Looks Like
After the test, the engineering team had a complete per-service saturation map derived from real test data rather than theoretical estimates.
| Service | Avg Response Time | Assessment |
|---|---|---|
| device-token | 1.5 ms | Well within limits |
| playback-auth-service | 137 ms | At ceiling at peak |
| heartbeat-service | 2.4 ms | No issues |
| user-catalog | 379 ms | Ceiling hit, resiliency triggered |
| widevine-proxy | 82 ms | No scaling issues |
This is not a test that revealed weakness. It is a test that revealed exactly where the ceilings are, which services need headroom adjustments, and what the platform looks like under conditions it had never been subjected to before.
A P95 spike that stays isolated to one non-critical service while P50 holds at 4ms is not a failure. It is the architecture working as designed.
.webp){kind=small}
