# Soak Testing: Finding Memory Leaks and Slow Burns > Soak testing (also called endurance testing) runs sustained load for hours or days to surface memory leaks and other issues that take time to appear. Source: https://loadster.com/guides/endurance-testing/ ## Introduction Some performance problems hide in plain sight. A web application that passes a 15-minute load test can still run out of memory after 6 hours, fill a disk after two days, or gradually exhaust a connection pool until requests start timing out. Short tests don't catch these because they don't run long enough. Soak testing (also called endurance testing) fills that gap. You apply steady, moderate traffic for a long stretch of hours or even days and watch what the system does over time. The goal isn't to push the system to its limit... It's to confirm that a system staying online under normal use doesn't gradually fall apart. ## What Is Soak Testing? A soak test is performed with a [load testing tool](https://loadster.com/compare/load-testing-tools/) like any other load test, but for a longer duration so that time-sensitive failures can surface. Unlike a stress test, which pushes a system past its capacity to see where it breaks, a soak test holds load at a realistic level and waits. Problems that only show up after hours or days of sustained load — memory leaks, storage exhaustion, gradual slowdowns — get a chance to appear. The name comes from the idea that the system is "soaking" in load: steady pressure, no spikes, long duration. The term "endurance test" means exactly the same thing. Some teams also use "stability testing" for the same concept, though stability testing can sometimes imply a broader scope (including functional tests over time, not just performance). ## Why Soak Tests Matter For Your Site There are a few classes of problems that may never show up in a short test but can take a production system down within a day: - **Memory leaks.** The application holds references longer than it should and the heap grows steadily. A short test looks fine; an overnight test eventually runs out of memory or triggers garbage-collection pauses. - **Connection pool exhaustion.** Connections to a database, message broker, or third-party API leak one at a time until the pool is drained and requests start failing. - **Disk-fill issues.** Log files, cached artifacts, or temp files accumulate faster than they're rotated or cleaned up. The system works fine until the disk or partition is full, then everything fails at once. - **Cache bloat.** In-memory caches without a proper eviction policy grow unbounded. Performance degrades slowly as lookups scan more entries. - **Background job backlog.** Scheduled jobs run slightly slower than incoming work. Each job completes, but the queue grows. - **File descriptor leaks.** Sockets, files, or pipes aren't closed and slowly accumulate toward the process limit. These problems all have something in common: they don't exist initially but show up after a certain amount of time has elapsed. To detect or reproduce them, the test has to run long enough to observe the effect. ## Soak vs. Load vs. Stress vs. Spike Testing These techniques all simulate traffic but answer different questions. In short: - **[Load testing](https://loadster.com/guides/load-testing/)** tests whether the system can handle expected traffic. This typically takes just 15–60 minutes. - **[Stress testing](https://loadster.com/guides/web-application-stress-testing/)** tests what happens when traffic exceeds capacity. This ramps load fairly aggressively until the system breaks. - **[Spike testing](https://loadster.com/guides/spike-testing/)** tests how the system handles a sudden, sharp burst of traffic. - **Soak testing** tests whether the system stays healthy over hours or days at realistic traffic levels. A thorough performance engineering program runs all four types of performance tests. See [Load Testing vs Stress Testing](https://loadster.com/guides/load-testing-vs-stress-testing/) for a longer breakdown of the two most commonly confused variants. ## How to Run a Soak Test A soak test has the same anatomy as a load test, just held steady for longer. **Duration.** At least several hours. Overnight (8–12 hours) is a common baseline; a full weekend run is a more thorough check for slow-climbing metrics. For systems with long-running scheduled jobs (nightly reports, daily rollups), you want the test to cover at least one full cycle. **Traffic level.** Representative of production, not peak. A typical rule of thumb is 50–80% of a stress-test ceiling, or whatever matches your busiest-typical-hour traffic. The point is steady sustained pressure, not a maximum. **Scripting.** Reuse your existing load test scripts. The scenarios that matter for a soak test are usually the same ones you already run for load tests — the user flows that keep the system warm — so a separate script library isn't typically required. **What to monitor.** Throughout the test, watch: - **Memory** — heap usage, RSS, or equivalent. Trend matters more than the number. - **CPU** — should be roughly flat once the system is warm. - **Open file descriptors, sockets, threads** — these shouldn't climb over time. - **Connection pool utilization** — database, HTTP, cache. Leaks here cause the most dramatic failures. - **Disk space and log file sizes** — especially if you don't rotate logs aggressively. - **Queue depths** — for any background work. - **Request latency and error rate** — are they flat, or drifting up? Loadster handles the traffic side of this; your APM or infrastructure monitoring handles the target-side metrics. A solid soak test usually has both windows open side by side. ## Common Issues Found During Soak Tests The issues soak tests catch tend to fall into three buckets: 1. **Leaks.** Memory, file descriptors, database connections, thread pools. These usually show as a metric that climbs steadily without stabilizing. 2. **Accumulation.** Logs, caches, queues, temp files. Same shape — climbing — but the fix is usually rotation, eviction, or rate-limiting rather than freeing resources. 3. **Scheduled job interference.** Nightly database vacuums, backups, or report jobs that coexist fine at 3 a.m. but collide with sustained traffic when they run during an endurance test. In most cases, the fix is cheap once the issue is identified — a missing `close()`, a misconfigured cache size, an unrotated log. The value of soak testing is in surfacing the issue before it takes down production, not in fixing anything exotic. ## When to Run Soak Tests Soak tests take longer than other performance tests, so most teams don't run them on every build. Reasonable triggers: - Before a major release, especially one that changes long-running behavior (new caching, new background jobs, changes to connection pooling, new memory-intensive features). - Before a seasonal peak or extended campaign where traffic will stay elevated longer than usual. - After a production incident that hinted at a slow-climb issue, to reproduce and verify the fix. - On a regular cadence, like monthly or per-release, for critical production systems where quiet regressions are expensive. If you've never run one, the first soak test will likely find something interesting. ## Conclusion Soak testing is the cheapest and safest way to catch problems that grow with time. A short load test tells you the system handles the traffic; a soak test tells you that it keeps handling the traffic. Both are useful, and they answer different questions. If your performance program only runs short tests, you'll miss an entire class of bugs that will eventually emerge in production instead. ## Frequently Asked Questions ### What is soak testing? Soak testing (also called endurance testing) applies steady, moderate load to a system over a long period — typically hours or days — to surface slow-burn problems like memory leaks, resource exhaustion, and log file growth that don't show up in short tests. The goal is to confirm the system stays healthy under sustained use, not just under brief spikes. ### What's the difference between soak testing and endurance testing? They're the same thing. "Soak test" and "endurance test" are used interchangeably in performance engineering to describe a long-running test at moderate load. Some teams also call this "stability testing." The technique is identical regardless of the name. ### How long should a soak test run? Long enough for the slow-burn issues to actually surface. A few hours is a reasonable minimum; many teams run overnight (8–12 hours) or across a full weekend. Memory leaks typically show up within a few hours, but log rotation problems, disk-fill issues, and scheduled-job backlog may only surface across days. ### What load level should a soak test use? Moderate, not peak. Soak tests use a sustained load that represents typical production traffic, or a little above it. The point isn't to push the system to its limit — that's stress testing — but to keep steady pressure on it long enough for gradual degradation to appear. ### What should I monitor during a soak test? Memory usage, CPU, open file descriptors, database and HTTP connection pools, disk space, log file sizes, and queue depths. Watch the trend over time, not just the instantaneous value. A metric that climbs steadily without plateauing is usually a leak. ### When should I run a soak test? Before any release that materially changes long-running behavior — new caching, new background jobs, new database pooling, or a new memory-intensive feature. Also before a major campaign or season where traffic will stay elevated longer than usual. For critical production systems, many teams run a soak test every release as part of performance regression testing. --- **Next Steps:** Ready to run a soak test of your own? Try [Load Testing with Loadster](https://loadster.com/use-cases/load-testing/) to create soak-test ramp patterns, multi-region load tests, and live metrics that surface the slow-burn regressions soak tests are designed to catch. #### Related Guides * [Load Testing Guide](https://loadster.com/guides/load-testing/) — primer on load testing types and when to use each. * [Load Testing vs Stress Testing](https://loadster.com/guides/load-testing-vs-stress-testing/) — the difference between the two disciplines. * [Load Testing Best Practices](https://loadster.com/guides/load-testing-best-practices/) — getting good results from any performance test. * [Best Load Testing Tools](https://loadster.com/compare/load-testing-tools/) — comparison of open source and commercial options.