Guides »

Scalability Testing: Scale Up, Find Bottlenecks

By Andy Hawkes
Reading Time 7 Minutes

Introduction

As a site or web application’s usage grows, your team eventually needs to know its capacity and what will happen when traffic doubles.

Adding servers might increase capacity, but the improvement is rarely perfectly linear because those servers often depend on shared databases, caches, queues, and external services.

Scalability testing measures how performance changes as you increase both the load and the available resources. The results help you understand how much capacity a scaling change actually provides and which component becomes the next limiting factor.

What Is Scalability Testing?

Scalability testing measures how a system’s performance changes as the load increases, and whether adding resources actually increases capacity.

Rather than only confirming that the system handles today’s traffic, scalability tests measure whether the system can grow by scaling up or scaling out to meet projected future demand. The result is a comparison of response times, throughput, and errors across several load levels and infrastructure configurations.

A baseline load test asks a bounded question: does the system perform acceptably at an expected level of traffic?

Scalability testing asks what happens as traffic continues to grow and whether additional resources move the capacity ceiling. In practice, a scalability test is usually a series of comparable load tests run against different infrastructure configurations.

Scaling Up versus Scaling Out

Scaling up, or vertical scaling, means moving to a larger machine with more CPU, memory, or faster storage. It can be the simplest way to add capacity, but every platform has practical and financial limits on machine size. A scalability test measures how much improvement each size increase actually provides.

Scaling out, or horizontal scaling, means adding more machines behind a load balancer. Many web application tiers are designed to scale this way. However, the additional servers usually share databases, session stores, caches, and other stateful services. One of these shared components might become the bottleneck even when the web tier still has room to grow.

Horizontal autoscaling controls when additional instances start, but it doesn’t guarantee that the rest of the architecture can use them effectively. If the database is already saturated, starting more application instances might add cost without increasing throughput at all!

Whether you plan on scaling up or scaling out, you can’t always assume your application will scale linearly. It almost certainly has architectural bottlenecks or shared resources that prevent scaling past a certain point.

Running scalability tests is the best way to find out.

How to Run a Scalability Test

Step the load up in stages

A typical scalability test repeats the same scripted user behavior at a series of increasing load levels.

For example, you might run a load test with plateaus at 100, 200, 400, and 800 concurrent bots. Hold each level long enough for response time, throughput, and resource utilization to stabilize before increasing the load. Several minutes might be enough to test a simple system, while autoscaling or slower background processes could require longer plateaus to get clean measurements at each level.

In Loadster, load test scenarios support flexible ramps and stages, so a single bot group can climb through a stair-step profile with its own ramp pattern, target bot count, and duration at each stage. Each bot represents one concurrent user running your script on repeat.

Change the resources and run it again

Next, change one part of the infrastructure and repeat the same stepped profile. You might add an application server, upgrade the database instance, or enable autoscaling. Compare the test reports side by side to see whether the change increased capacity or simply moved the bottleneck to another component.

Keep the scripts, load stages, test data, and other environmental conditions as consistent as possible between runs. If several variables change at once, you won’t know which one caused the difference in performance.

Reading Scalability Test Results

Start by comparing throughput with the bot count. While the system has enough capacity, doubling the bots might approximately double the number of pages or transactions completed per second with little to no performance degradation. When throughput begins to level off, response times might rise and errors might appear. At that point, you’ve probably reached the scalability limit of the current configuration.

Correlate this point in the test with your server and application metrics to identify which resource saturated first. It might be CPU on the application tier, database connections, memory, or network bandwidth. The guide to performance tuning web applications covers common bottlenecks to investigate. Rerun the same stepped test after each change to confirm whether the capacity ceiling moved.

Once you’ve determined your system’s scalability in various configurations, you can use these results to create a basic cost model.

Let’s say you discover that each additional application server supports approximately 300 more concurrent users before another shared component reaches its limit. You can estimate (with caveats) how many servers next year’s projected peak requires and what that configuration will cost. That forward-looking process is covered in more detail in the capacity planning guide.

You might also find out that your architecture scales out linearly up to three application servers, but gets no extra benefit from a fourth, because the database gets maxed out. In this case you might need a combination of horizontal scaling at the application server layer (more app servers) with vertical scaling at the database layer (a bigger database) to continue growing.

In either case, it’s better to find out through scalability testing and proactive planning than by flailing around in production during a traffic spike.

When to Run Scalability Tests

Run scalability tests when your projections show that a marketing campaign, seasonal peak, or organic growth might exceed current capacity. Scalability testing is also useful before committing to an architectural change meant to improve scaling, or when comparing instance sizes and hosting configurations whose costs are easier to predict than their actual capacity.

Scalability testing can also explain unexpected autoscaling behavior. If the number of instances continues to grow while throughput and response times don’t improve, a controlled round of scalability tests can help locate the shared bottleneck.

Conclusion

Scalability testing measures how much capacity each additional resource actually provides and identifies the component that prevents further growth. Run the same stepped load test against several controlled configurations, compare the results, and use the measured scaling curve for architecture, capacity, and budget planning.

Frequently Asked Questions

What is scalability testing?

Scalability testing measures how a system’s performance changes as load increases, and whether adding resources (bigger machines or more machines) actually raises capacity. Rather than confirming the system survives today’s traffic, it verifies the system can grow to meet increasing demand, and finds the bottleneck that stops it.

How do you measure scalability?

Run the same scripted user behavior at increasing load levels, like plateaus of 100, 200, 400, and 800 concurrent users, and graph throughput against load. In a healthy range, doubling the load roughly doubles throughput. The point where throughput stops growing linearly and response times climb instead is your scalability limit for that configuration.

What is the difference between scalability testing and load testing?

A load test asks a bounded question: does the system perform acceptably at an expected level of traffic? Scalability testing asks an open-ended one: what happens as load keeps growing, and does adding resources move the ceiling? A scalability test is usually a series of load tests run against different configurations.

What is the difference between scaling up and scaling out?

Scaling up (vertical scaling) means moving to a bigger machine with more CPU, memory, and faster disks. Scaling out (horizontal scaling) means adding more machines behind a load balancer. Most modern web applications grow by scaling out, and scalability tests usually reveal that shared resources like the database are what limit it.

Why doesn't doubling servers double capacity?

Because somewhere in the stack a shared resource refuses to double along with them. Databases, session stores, caches, and load balancers serve every node at once, and one of them usually saturates first. Finding where the scaling curve flattens, and which component is responsible, is the main point of a scalability test.

Next Steps: Want to find your scaling curve? Try Load Testing with Loadster to build stepped load scenarios, run them against different configurations, and compare the results side by side.