— We will code tests later...
— Sorry, I’m not going to buy it.
Probably you are a programmer, a chief or a customer. The most important thing is that you are connected with software development, and just today you said it yourself or heard from someone something like that:
- Now, we are developing the code, but we will develop tests later since currently, everything is changing. But as soon as changes stop, we will implement them immediately.
- Let’s skip test design because it will slow us down. In this case, we will be able to move forward much faster.
- Unfortunately, we don’t have any time for implementing tests, so we will develop only code… If we have time, we will do it.
There are many variations, but a result is almost always the same – later or never. When we are at the beginning of our career path, we are young, full of strength, self-confident, our judgment isn’t clouded, we are steeled by struggling with university laboratory researches, on which we worked all last days and maybe even weeks. It seems like the sky’s the limit for us. And we code so fast, that nothing is necessary but paper and that’s it. What tests, I beg you. It’s much faster to do it on your own rather than explain how to do it. Neglect of testing is growing inside: “It’s for losers, I don’t need it as I already develop it perfectly without it.“
Unfortunately, a real world is completely different. In a real world, you spend months, years creating software, but our bad habits, acquired over the years of laboratory researches, don’t just disappear. As a result, newcomers become professionals but adopt the same approaches, applied for simple applications that will be thrown away tomorrow.
In this article, I’ll try to consider how automated testing influences the result we get today, tomorrow and in a year. I’ll try to consider typical wrong beliefs about automated testing and make an argument for using it. This whole article is a result of the author’s experience and misbeliefs, who hopes readers can do a critical review of the information. Mostly, everything, described in this article, is related to unit, integration and smoke tests.
So, first, we will look at an interesting picture that, strangely enough, isn’t obvious for many people in our field. It shows new features delivery velocity into software depending on project duration.
This picture shows a lifecycle of most software projects although its parts can differ a bit from project to project. Let’s consider what is happening more closely.
First, a project was developed by some super developers (Level of “God”), who were very smart and qualified. They had some godlike plan that led to the fast acquisition of new functions by the project (Gods Era).
At some exact moment, good strong developers were recruited into a team who started system learning and distracted guru from vital concerns so fiercely. Adoption Era started during which new features appeared more rarely as developers most of the time tried to understand how they should develop this godlike creation (often without any records and grudging support of Gods who were reassigned to another project).
Making mistakes over and over again, the team gained insight and managed to develop code, adding new functions. Renaissance Era began during which everybody was happy, a product grew, became stronger and customer managers got perks for implementation.
However, in the course of time employees started leaving the company. Some of them got bored to code in Java 5 (It’s already Java 9 outside), some were reassigned. As a result, the team has the decline of competence. And the product became so complex that every new function implementation caused plenty of changes. Decadence Era started.
Sadly, many products will face such a lifecycle. Reasons can be absolutely different – from a lack of team qualification till completely changing business demands. As a rule in good businesses, on the bones of such a product a new one is born, where all previous mistakes are taken into consideration and new ones are made.
In terms of our article, two last stages are interesting – Renaissance and Decadence. Teams put (or must put) a lot of efforts to postpone Decadence, prolonging Renaissance. There are many ways to do it: design pattern, dependency inversion, code reuse, reduction in components size and weak cohesion, better code and interface documentation, code training for employees, automated code testing and other practices of better software creation.
Possibly, it isn’t immediately obvious, but automated testing plays some specific and the most crucial role among best practices. And now we take a look at the reasons. Unfortunately, this is development and test automation frameworks support that most of the developers dislike. They are not capable to develop these tests right and even may consider such an activity as humiliating and enforced.
Quality of a software product
To understand why automated testing is the most crucial component among many practices of Renaissance prolonging, we should start with understanding what quality is. The common meaning of “quality” is “meeting expectations”. The question is: expectations of whom? An answer is easy – expectations of all interested parties: a developer, a QA engineer, a team leader, a project manager, a product manager, a product owner, a user. Quality is essential for everyone, but everyone understands it differently. Let’s look at an example that can be up-to-the-minute for participants of some project.
A product user
- No bugs,
- performs fast,
- user friendly,
- profitable.
A product owner
- Meets the requirements,
- no bugs,
- users are satisfied with it.
A product manager
- Meets the requirements,
- no bugs,
- meets product owner’s expectations,
- meets users’ expectations.
A project manager
- Meets the requirements,
- high test coverage,
- all builds complete successfully,
- regression testing doesn’t discover repeated defects,
- tests don’t discover defects in untouched components,
- coverage of all use cases with smoke tests.
A team leader
- Unit tests are implemented in a qualitative manner and provide high coverage,
- integration tests are implemented in a qualitative and comprehensive manner,
- smoke tests are integrated into CI pipeline and are performed with other tests,
- tests complete successfully when there is a merge request,
- develop and master branches build and deploy successfully,
- a code is implemented with using the best software development practices,
- all merged changes passes reviews successfully.
A QA engineer
- A test plan is thorough and up to the minute,
- regression test complete successfully according to a test plan,
- automated tests are performed successfully while integration,
- high automated testing coverage of a test plan.
A developer
- Code meets the standards of a team,
- code is implemented due to an assignment,
- classes and methods are covered with unit tests
- integration tests are implemented for complex interaction,
- no issues from a PR reviewer when code merge is requested,
- when performing CI/CD go through all check stages in an integration system.
The perception of the quality by participants can differ and include other criteria which aren’t shown in this table. In reality, different participants are interested in quality to different extents: some require considerable ways to measure quality, other perceive it as the result of product success. However, there are such people who create this quality. These are developers. Ironically and coincidentally, because of lack of time, the presence of laziness and overconfidence and due to other reasons, the developers consider their product of a good quality just because it is created by them even factoring that they don’t apply any practices, methods, assessment instrument and quality improvements. In a nutshell: “The product is of a good quality because I’m developing it”.
Fortunately, among developers, there are quite enough reasonable and sensible people who aren’t so overconfident to support this idea. They see it from another perspective: the perspective of responsibility, deadlines, productivity, spent time on a problem analysis and changes. More frequently, these are senior developers who developed a good deal of projects to realize that even the most virtuous initiative without any strict discipline leads to bad results. If there are such people in a team, it is highly lucky because its chances for success are high and life in Renaissance Era can be really continuous. If there are no proponents of strict discipline in developing, it’s likely a project plummets to Decadence Era very fast, developing becomes very slow, product quality is seen as low by all participants and budget doesn’t meet the expectations.
Connection of testing and team performance
Everybody must have noticed that if you need to repeat a state of some complex system in reality, after its decomposition, it can be quite complicated. To overcome this difficulty, we take pictures, collect data and use other tools for memorizing because our brain can keep simultaneously 7 elements, give or take 1-2 elements. So, when developing software, we can’t keep in our heads all the system and its interconnections which form a graph that isn’t always simple. Changing something at one place we influence the interconnected components, the part of which is beyond of our attention.
Consequently, there are two main reasons of bugs:
- wrong solution that led to a bug;
- right solution that led to a bug because not all interconnected elements were changed properly.
For example, a variable contains time in seconds. It was converted into minutes but interconnected components still see time in seconds. As a result, the whole system work is flawed. Therefore, quality worsening is a direct consequence of made changes in the system. It‘s impossible to make modifications and not to worsen the quality. Any changes cause temporary loss of quality. The point is that:
- where exactly quality loss happened;
- how to recall the quality to a previous level;
- how much time it will take.
Awareness of these questions determines the duration of Renaissance Era while software developing.
Let’s imagine a simple variant: a developer develops a code, QA engineer checks functions, regression testing is run once every two weeks. Let’s consider a delay in making modifications in the product.
Consequently, in some particular case when changing some component, it can take up to 3 days factoring 2 testing cycles and one fixing cycle. Plus 2 days to fix bugs found during regression testing of other interconnected components. The worst case scenario is that this cycle takes several weeks, when a simple function causes several iterations of “development – testing – regression testing” to resolve the issues. Let’s imagine that during iteration several components are changed, new ones are created. All of them have impact on each other that leads to likely big quality deterioration within a short time and the quality doesn’t return to a previous state so soon.
Of course, the example above is a bit marginal. As a rule, programmers are quite clever and use practices of coding which mitigate bugs. But the rule of “7, give or take 1-2” still exists: some developers make more mistakes, others less. Nevertheless, even the most talented make mistakes.
In the example, you can see that only testing of a component alongside with regression testing, which checked connected components, gave the opportunity to find bugs and fix them. So, only testing is able to answer the question:” Where quality loss happened?” Consider that it answers not only about conformity of quality before delivery to a user, but it is also important as a tool that helps to understand where this quality started going down.
In the previous example testing was integrated into development and was run for each component, it gave the opportunity to identify quality deterioration fast to quickly apply corrective actions. Now, imagine that testing is performed only before delivery of the project code to a customer. And sometimes it really happens: acceptance testing is run only at the end, which discovers problems (not all). It’s obvious that products, in which such an approach is applied, don’t demonstrate high quality.
Of course, it would be perfect if zones of quality regression were identified when occurring as soon as possible to discover not only the exact place where this bug is but the influence on connected components as well. Imagine that a developer just can’t integrate his modifications into a product because of low quality that has evidence. Only extensive automated testing allows reaching such a result.
Complex automated software testing
Complex automated testing covers next software levels:
- Separate classes, methods, functions – unit tests;
- Modules, interconnected objects – integration tests;
- Use cases – smoke tests (E2E tests).
Unit tests and integration tests must be created by developers because they reflect internal software architecture. Smoke-tests (E2E) are developed by QA engineers and reflect the perception of a product by real users (external specification conformance).
When using the best practices, all the tests described above are integrated into the common test framework that allows determining a current quality state of a product in time. Struggle for saving product quality is shown in the next picture:
We can see dips in a quality level and a correction zone, after which quality remains at a stable level till next integration of modifications into a product. The more rarely integration occurs, the deeper a dip is. The fewer tests are written, the longer recovery is. If there is not enough time for recovery, quality will continuously go down.
As practice shows, development through extensive automated testing obligatorily includes next objects:
- unit tests are developed and supported for all classes;
- integration tests are developed and supported for components (constellation of classes);
- if a bug is detected, specific unit tests and integration tests are developed for this bug;
- for each use case scenario e2e test are developed;
- for every found bug e2e test is developed.
Unfortunately, the more complex a test is, the more time it takes to pass the test. This is why all tests launch can take hours even using the high-performance equipment. There are various strategies to handle this issue, for instance, limitation rules for code modification, decomposition of independent libraries to exclude unplanned influence. Description of these approaches isn’t connected with this article.
Work with misbeliefs about automated testing
It’s impossible to list all excuses and arguments that opponents of automated testing will make but we often face next cases in our practice.
“We don’t have money for it”
Having read the article, you have already understood that it’s a huge mistake. A customer just doesn’t understand how it works and this will lead to the fact that development will be inefficient, the project will quickly move to Decadence Era and speed of new functions adding will be slow regardless of the engineers’ efforts.
“We release a project, It’s necessary to cover code with tests”
Thank God, you have managed to finish the project and now you are going to make your customer happy, strengthening his trust in your product. In this case, tests are implemented for tests’ sake. It gives no quality and tests are developed to provide beautiful coverage (pictures) but aren’t suitable for detecting quality regression.
“Tests will slow us down. We will to move forward much faster without them”
This is one more misbelief. If you had read this article attentively, you have already realized that everything will be just the opposite: in 1-2 months a project will turn into torture, a giant with feet of clay and to analyze zones where quality regression is will take more and more time.
“Now, everything changes so fast. When we stabilize architecture, we will implement tests”
To translate it into the human language: We will work now as we like and after we will code tests for what we have done to make you happy. Don’t buy it under any circumstances.
In fact it’s impossible to develop tests “later”
It’s likely there are other arguments made by opponents of automated testing. I’d say that only one of them is worth noticing: we are not capable doing tests. This is the argument you must respect and help your team to overcome this problem, help your team to learn it. It’s well known that the more you have, the more you want. Some people just can’t start developing the tests till somebody helps them to live in this environment, for example, holding TDD (test-driven development) workshop. There is a reason. If a developer has never developed software using testing, his code just isn’t suitable for testing.
In other words, such a code exists which isn’t possible to test at a high-quality level. Only development taking into consideration testing gives the opportunity to develop a code that is possible to test. A development approach, for example TDD, defines a code design. When a developer needs to create unit tests, integration tests, he has to think how to develop such a code that will be easy to test. Eventually, in such a code the best practices start occurring: compact functions, immutability, dependency inversion, avoiding of global objects. This is the consequence of the fact that codes must be tested. In fact, this is a unique phenomenon that is as exact as a philosophical maxim “Being determinies consciousness”: “Good tests improve the code design”. Consequently, due to better design, tests from a quality loss determiner tool transform into a quality improvement tool.
Probably, some part of the audience considers this article to be trivial and banal, especially when automated testing is an everyday work process. However, my experience of collaboration with different interested parties while developing software shows that these thoughts can be useful for people who look for pros of implementation of automated testing into their product. I hope it will help you to reconsider the perception of these practices if you aren’t a proponent of them or inspire you if you implement these tools into your project.
Read more about software quality and processes:
Please, share the post with friends if you like it.