Testing theory is a separate, important, and very interesting science in the field of information technology. Testing professionals are as valuable in a team as developers, DevOps, project managers, and other roles. They are responsible for ensuring the quality of software products and systems.

If you want to dive into the details of this theory, you can start with this link. There, testing is described quite well, what it is, what types it can be, and so on. But in this article, I will try to briefly explain the essence of testing and explain what we will strive for in ensuring the quality of our hypothetical software product.

So, we all know that software contains bugs. And the fewer bugs, the better. But what is a bug, and what does it mean for a system to be error-free? For myself, I have derived the following definition.

A system is free of errors and bugs if it:

• on the one hand, behaves globally exactly as the system authors intended;
• on the other hand, in detail, behaves as expected by the user.

In other words, the features (business logic) work as the authors want, and the UI and small details of interaction with the user work as the user expects. This should not be confused with UX design. I mean basic things, like pressing the "open file" button opens a dialog box with file selection, in the right place on the screen, with the right names and filters.

Now the question arises, how do we ensure that the system contains as few bugs as possible? Of course, by testing it. But how to test, when to test? Let's start with the question "when." I really like the approach called Shift left.

It is very simple - test as much as possible and as early as possible. Why? Because the "cost" of fixing a bug increases significantly over time it exists.

When a developer checks the code they write, errors are corrected very easily. Creating unit tests requires additional effort, but fixing errors after running such tests also "costs" not much (usually the author of the code writes unit tests while still in the context of the task). But already at the stage of functional and system testing, the "cost" begins to increase significantly, because other people are involved in finding such bugs, special frameworks are deployed, application deployment is done, and so on. This all takes time and money. Well, fixing a bug that has gone into release can be extremely expensive, as it can directly affect business processes and potentially deprive the company of some amount of profit.

In my mind, I divide the process of error detection into two major stages:

• tests written by developers
• tests written by QA teams (quality assurance)

Typically, in addition to testing code during its writing process, developers are also responsible for unit testing. Tests that verify low-level modules, methods, functions, and procedures usually require a deep dive into the code context, and if these tests are not written during or immediately after development, the complexity and effort required to write such tests increase significantly. This is because even after several days, the developer will need to immerse themselves in the code again, and someone who did not write the code will also need to study it to understand how to test it.
Therefore, I conclude that developers should be the ones to write unit tests. And if these tests are not written immediately, it's better not to write them at all and focus on testing at a higher level. In my opinion, this approach would be much more effective in this situation.

Next, the Quality Assurance (QA) team comes into play. Their work differs significantly from the tests written by developers. They test the high-level interaction of modules, the functionality (business logic) of the application, and user interaction but do not engage in low-level code testing for the reasons I outlined above. This team includes both manual testers and people involved in test automation. Who exactly is responsible for integrating into CI pipelines is an organizational question. In our case, let's assume it's a DevOps engineer responsible for the entire test automation pipeline.
The manual testing team studies the documentation and functionality of our software product and writes step-by-step scenarios for properly testing the system. These scenarios are then used for both manual testing and automation.

Part of the team responsible for automation creates tests, which I also divide into two parts:

• Developer tests
• System tests

Please note, that the term "system test" here has a slightly different meaning than is commonly understood in testing theory.

Developer tests are tests that work in an isolated part of the application. Technically, they are the same as unit tests, created according to the same rules and using the same frameworks, but they test not low-level module code but higher-level interaction within the application (integration tests) and, if possible, the correctness of the system's core functionality (functional tests). Essentially, in the code, one can emulate a specific user request and check how the system responds to it. Just without using the UI. These tests, like unit tests, can be run without deploying a full installation of the system. They are usually very fast and do not require special resources to execute.

System tests are any tests that require a full deployment of the system instance. These include API tests, UI tests, load tests, and others. They usually use special frameworks and require more time to execute. Consequently, the "cost" of running such tests is higher.

In the end, we get two categories of tests:

• fast and "cheap" but more superficial and synthetic
• slow and "expensive" but deeper and more comprehensive

Following the Pareto principle, we can expect that fast tests will help us detect about 80% of bugs, while slow tests will help find the remaining ones.

And based on this logic, to significantly improve the quality of our software, it's enough to run fast tests, for example, with each pull request, while running slow tests less frequently and on special occasions (for example, when preparing for releases).

And as a result, we get the following scheme

"Manual" testers write test scenarios that are automated for both "system" tests and "developer" tests. Both types of tests are enriched with integration and other types of tests at the discretion of the team and integrated into CI pipelines. We run developer tests as often as possible, for example, at the pull request level. This allows us to catch most errors at early stages when they can be fixed by developers working on the pull requests. System tests are only run when the entire system is built, for example, before any releases, and they help to identify higher-level bugs (such as those related to UI or system environment).

It is precisely for this approach that we will write a YAML pipeline for Azure DevOps in the next article.