Boost IT reliability with synthetic testing

Synthetic testing is the sum of the operations and processes used to create synthetic tests. These represent a key method of gaining insights into how users interact with your applications. What is synthetic testing is often answered as if it were the same as synthetic monitoring, which comes close but isn’t quite right.

Synthetic tests are the core of synthetic testing. These tests are created to simulate the user's behavior in an application, or over a network, or using an API, and are invaluable in checking for possible issues, preventing disruptions, and identifying pain points in the final user experience. Synthetic tests are written to generate synthetic data that is in turn used to test reliability, discover bugs, and check the performance of services or assets. The tests are often manual, but can be automated. The process of automating these synthetic tests and analyzing them is correctly referred to as synthetic monitoring, which takes care of running the tests and continuously monitoring their results. Thus, synthetic testing and monitoring are not quite the same but always operating in tandem.

Synthetic testing plays a crucial role in identifying performance and experience issues by replicating genuine traffic. This proactive approach is fundamental in helping companies discover and prevent availability and performance problems. This is achieved through various platforms for developing synthetic tests, with their own syntax and running steps. These platforms may or may not be integrated into a synthetic monitoring tool.

One of these platforms is the Robot Framework, which comes with libraries for many programming languages upon which integration tools can be developed. This is the framework that Checkmk’s Synthetic Monitoring add-on uses, and how Checkmk can make use of the proactive monitoring approach of synthetic tests.

Broadly, there are two main applications that are most common for synthetic tests: browser and API tests.

Within browser tests are all those which manipulate the web interface of an application: clicking buttons, following links, filling out forms, adding items to charts (for e-commerce apps), signing into and out of accounts, authentication, and much more. Everything necessary for a functional web application can be tested via synthetic testing. Libraries offered by synthetic testing frameworks can emulate a browser (and multiple versions of one) to test how a hypothetical user would accomplish various tasks with it. Via these libraries, it becomes easy to test various browsers and all their versions to ensure perfect compatibility of your application with the desired ones. Furthermore, logical fallacies in the UI and usability issues in the web app can be quickly discovered before they affect real users.

API testing instead includes a few subcategories of synthetic tests. Most commonly, testing API by making HTTP requests to check for availability, page load times, status codes, and header or body content are developed in a synthetic testing setup. Next, conducting a few SSL tests to check if users can correctly authenticate with an API by validating SSL certificates is a very typical use case of API testing.

Going back one step, using TCP-based tests is useful for checking the availability of critical service ports such as SSH or DNS. If they aren't available, it may signal a server-side issue that needs to be investigated. Similarly, DNS requests are made to check their response times and detect potential configuration errors.

Back one step further, simpler ICMP-based tests can be used in synthetic testing to test network delays and basic connectivity issues. All these may also be chained together into what are called "multistep API tests" to see how data is passed from one API endpoint to another. Any interruption or tainted data is a possible signal of a deeper issue that needs to be solved before the API can be considered fully functional.

Of course, both browser and API tests are not to be run in a vacuum; they can be run together, sequentially, or in parallel. Both give the administrators and testers insight into what is working and what is not, or potentially will not. These tests are better combined into a synthetic monitoring software, like Checkmk with the Synthetic Monitoring add-on, to provide a complete view of your infrastructure together with the real user data coming from classical monitoring checks.

Synthetic testing is useful in a vast number of cases. With the power coming from tests, synthetic data can be created to replicate real-world conditions that have not happened (yet) in your environments. For instance, a larger volume of traffic than you usually detect on a network, or different API requests than what you have noticed so far, etc. It is easy to see how synthetic testing is important for quality assurance (QA), not just to detect issues in production.

Enabling the creation of any type of synthetic data and any quantity of it, synthetic tests can ensure scalability to different loads, across various locations. You don't need to guess how your application will react to hypothetical loads – a synthetic test can be used to test it. Real user experience can be tested before launch by automating the user paths on your application, and immediately spot issues and responsiveness, a key pain point for most users. This is called synthetic application monitoring, and it perfectly pairs with normal application monitoring, which uses real-world data once the application is launched or being beta tested.

Similarly, in API testing, it is invaluable to write some synthetic tests to thoroughly test your API before making it public. This enlarges the range of requests that you can send to the API and check how it responds, if it is appropriate, or if there are issues (even security flaws) in it.

These and more use cases make up an important aspect of synthetic monitoring – one that is fundamental to implement for any modern company.

One challenge of synthetic testing is its complexity. Whether you are testing an application, or checking the functionalities of your developing API, the sheer number of parts that can go wrong is daunting. Writing all the tests to check every area and possible issue is a monumental task that often makes administrators prefer real user monitoring of real data from production.

Further, creating synthetic tests often requires coding skills and knowledge of scripting languages. Non-technical team members may see it as too high of a barrier and revert back to passive monitoring solutions only. Setting up a series of synthetic tests for a single application or interface may require many human hours. Synthetic testing frameworks, like the Robot Framework, often help here by providing pre-written tests or plenty of examples to get the developers started in minimal time. Implementing these tests in a synthetic monitoring solution is often simplified and nearly automatic, like it is for the Synthetic Monitoring add-on for Checkmk. Still, some initial effort is necessary.

However, when tests are written and run, there's the risk of them being quite prone to breaking. For instance, moving or renaming an UI element may cause multiple tests to fail, causing plenty of false alarms. Occasional rewrites are a necessity for most synthetic tests. Technical debt is a factor of consideration when deciding on a synthetic testing strategy.