Changelog 2023.06

Manual Timers

Loadster automatically tracks timing metrics for any browser navigation (in the case of Browser Bots) or HTTP step (in the case of Protocol Bots). These metrics appear on the Response Times graph in the test report.

You can also add custom transaction timings to the report from a code block. Traditionally, it’s been done like this:

// report custom timing for everything executed in this block
bot.timer("My Custom Transaction", () => {
browser.navigate('https://example.com');
bot.wait(3000);
browser.click('button.join');
});

We’ve now introduced another way, in case you want to calculate the timings yourself.

// report custom timing in milliseconds
bot.timer("My Custom Transaction", 3150);

This might be useful if you have your own way of calculating timing. For example, you might want to combine the timing of multiple steps but subtract the wait times.

Custom Cluster Sizing

Loadster automatically sizes cloud clusters for your test based on the type and number of bots. For the large majority of load tests, the default cluster sizing works fine.

However, certain sites impose an unusually heavy burden on the client’s CPU. This mainly includes sites with lots of video, 3D graphics, or client-side performance bugs. When you’re load testing a site like this, default clusters may become overloaded. You’ll usually see it on the Engine CPU Utilization graph when the CPU utilization approaches 100%.

To help with this we’ve introduced custom cluster sizing, so you can request that your bots be launched on clusters with 2X, 4X, or 8X resources.

For now, the ability to request larger clusters is enabled on a per-account basis, so if you’re experiencing the symptoms of overloaded engines and would like to launch bigger clusters, please email help@loadster.com.

Wildcard Regions

The biggest load tests require thousands of cloud instances, and even the large AWS regions occasionally have intermittent capacity shortages that prevents the test from launching. A few of our customers have had to resort to trial and error – moving bots around and trying again with different regions – in order to get their test to launch successfully.

To avoid the uncertainty of false starts, we’ve introduced a “wildcard” region. When you select the wildcard region, Loadster will automatically choose a cloud region that has capacity for your test. This should make the process of launching unusually large load tests smoother.

Please contact help@loadster.com if you’d like the wildcard region enabled for your account. Note that this is usually only necessary for very large load tests with tens of thousands of Browser Bots, so if you haven’t experienced this you probably don’t need it.

Content Security Policies

Content Security Policies (CSPs) give sites a way to restrict which resources may be loaded onto the page. They are meant to prevent against certain Cross Site Scripting (XSS) and Click Jacking attacks.

As it turns out, some sites now have very strict CSPs that prevented Playwright (the browser automation library used by Browser Bots) from injecting JavaScript onto the page to take screenshots or perform other legitimate testing actions.

Loadster’s browsers will now launch with CSPs disabled by default, since you are load testing your own trusted site with bots. If you want to keep them enabled, you can toggle the “Disable Content Security Policies (CSPs)” setting in the Browser Bots section of Settings.

Additional Features & Improvements

• Added a browser.setUserAgent(userAgentString) method for when you need to override the User-Agent for Browser Bots. Previously it was possible to override the User-Agent header sent to servers, but this didn’t help with client-side JavaScript or analytics that interrogate the user agent.
• Made the “Next Steps” section at the top of a load test report show up for 30 days following the test. Previously it only lasted about 1 day.
• Tidied up the printable stylesheet for test reports a bit, although it still has room for improvement.
• Improved the log messages in the cluster log when a test fails to launch because one or more cloud regions had capacity problems.

Bug Fixes

• Fixed an inconsistency in the Errors by Type and Errors by URL counts for an edge case where errors arise from a code block.
• Fixed a UI issue with our custom dropdown component going offscreen when it contained a large number of values.
• Fixed a bug on the publicly shared test reports that sometimes caused the engine/region and bot type to show up incorrectly.
• Fixed a WebSockets race condition that occasionally caused the script editor to say a script had failed to play when it actually did play.
• Fixed an issue that caused bots to ramp up in a stair-stepping pattern instead of a smooth curve when a high cluster multiplier was used for the bot group. This was only noticeable in tests with Browser Bots using a high cluster multiplier, which was uncommon.
• Fixed an incorrect displayed row count that was displayed after uploading a very large CSV dataset.
• Made the silent option work for select operations, like browser.selectOption('#element', '[0]', { silent: true }). Previously this worked for most other element operations but not selecting an option from a dropdown.
• Adjusted API rate limits to prevent self-hosted engines from getting rate limited.