Concurrency Testing: Deadlocks & Race Conditions

Introduction

Web applications need to handle multiple concurrent requests. These requests might arrive at the same millisecond, or they might arrive close together and be processed in parallel. Some requests take your server longer than others to process. Trouble can arise when two concurrent requests both try to modify the same thing in your application, raising the possibility of race conditions, deadlocks, and other troublesome failure modes.

Your system can return every response quickly and still produce incorrect results. For example, an ecommerce site might oversell inventory, you could process the same payment twice, or you might lose one of two updates when several requests try to change the same data at the same time.

Concurrency bugs can be tricky to reproduce because they only happen when two or more operations execute in parallel and finish in a particular order. A single tester manually clicking through your application is unlikely to create that unfortunate timing, so concurrency testing tools are usually required.

What Is Concurrency Testing?

Concurrency testing is the practice of testing web applications under heavy concurrency to detect or reproduce concurrency-related bugs.

Concurrency testing checks for race conditions, deadlocks, duplicate processing, lost updates, and other correctness problems caused by simultaneous activity. While ordinary load testing typically measures response times, throughput, and errors under traffic, concurrency testing does all this but also focuses on ensuring the completed operations produced the right outcome.

Catching this sort of bug outside of production usually requires a load testing tool to generate the simultaneous activity and evaluate the results. The load test supplies the concurrency, and your checks on the data itself record the outcome.

Common Concurrency Bugs

Most concurrency problems boil down to two operations accessing or changing some shared state or resource at approximately the same time.

Here are a few common examples of concurrency bugs:

• Race conditions. Two users buy the last item in stock at the same time. Both requests check availability, both see one item remaining, and both purchases succeed. The store has now sold 11 units when only 10 were available. Someone’s going to be disappointed!
• Deadlocks. Two database transactions lock the same rows in opposite order. Each waits for the lock the other holds, and neither can proceed until the database cancels one of them. This might appear as an intermittent timeout or transaction error under load.
• Lost or clobbered updates. Two users update the description on a card simultaneously in a project tracker. The last write clobbers the earlier one, so someone’s work is lost.
• Duplicate processing. An impatient user clicks a payment button twice, or a client retries a slow request. Without an idempotency check on the server side, the application might process the same payment twice.
• Session or cache bleed. An incorrectly scoped cache key or shared session state might cause one user to receive data belonging to another user. This might be rare, but load tests with many users can make the problem easier to expose.

How to Run a Concurrency Test

Target the same contended resource

Unlike a general-purpose load test that distributes activity across many pages and resources, a concurrency test might deliberately direct many bots toward the same resource at the same time. In a Loadster scenario, the aggressive ramp-up pattern starts most bots quickly and makes this kind of simultaneous activity easier to reproduce.

Build your test scripts to use the same inventory item, seat map, account, or other shared resource. Datasets can give each bot distinct user credentials while all of them target that common resource or set of resources.

Verify the data afterwards

The verification step is what actually makes it a concurrency test. If you started with 10 items in stock, confirm that exactly 10 purchases succeeded and that the inventory never became negative. For financial operations, reconcile the number and total value of the completed transactions against the final balances.

Reconcile the counts from the load test report with the counts of the resulting transactions in your application and make sure they’re exactly the same.

In a load testing tool like Loadster, capping the bot group’s total iterations gives you a known transaction count to compare with your application data after the test.

Watch for the quieter signals

Deadlocks and lock contention often masquerade as timeouts or slow response time outliers during the period of highest concurrency in your test.

Make sure to check your database logs after the concurrency test, because many databases record canceled deadlock transactions even when the application retries the operation and eventually returns a successful response.

Your application logs might also show errors that indicate concurrency-related problems, but sometimes they are silent.

When Concurrency Testing Matters Most

Concurrency testing is important wherever there’s a chance that multiple users compete for the same resource. Typical examples include checkout and payment flows, inventory updates, appointment booking, reserved seating, ticket sales, and operations that update financial balances. If simultaneous access could cost your organization money or user trust, it’s worth including that flow in your concurrency tests.

Whether planned or unplanned, a traffic spike can bring a large flood of users to the same resource within a short period of time. Ticket releases, flash sales, big announcements, and appointment openings are good examples.

You don’t necessarily need a separate set of tools or scripts to conduct concurrency testing. The same tools and scripts that you use for load testing can often run a concurrency test. You’ll just want a fast ramp-up, a shared target resource or set of resources, and a careful data verification step after the test completes.

Running concurrency tests after significant code changes or before a large launch or sale can catch bugs that ordinary performance tests don’t.

Conclusion

Concurrency testing verifies correctness under simultaneous or parallel processing, which isn’t something response time and throughput measurements alone can pin down.

Use a load testing tool to carry out simultaneous operations, direct them toward some shared resource or state, and then verify the final data and counts are correct.

Especially if your application handles payments, inventory, bookings, or has other contended resources, concurrency testing should be part of your performance testing process.

Next Steps: Try Load Testing with Loadster to direct concurrent bots toward your most contended application flows and verify the results afterwards.

Related Guides

• Load Testing Guide — primer on load testing types and when to use each.
• Spike Testing — sudden surges, the natural habitat of concurrency bugs.
• Capacity Testing — finding your true maximum acceptable load.
• Load Testing Best Practices — getting good results from any performance test.