At funda, QAs are highly outnumbered by developers. So how do we test our products? Anıl Kuşaksız, Senior QA Engineer, gives 5 reasons why we at funda believe in testing as a team and shares some useful tips on how to implement it.
If you search ‘optimal QA vs. developer ratio’ on Google, you might find very different results. Suggestions start from having no QAs, to 30 developers to 1 QA, or 1 developer to 1 QA. The number highly depends on the sector and the type of company. If a company develops medical devices or autopilot software for an aeroplane, many QAs would be needed to test it properly.
Source: WarnerMedia Studios & Networks
We have different scrum teams with different sizes. There are teams with 3 developers to no QAs, and others with 6 developers to 1 QA. Having an army of QAs might give developers the assurance that ‘someone is going to test my code anyway’, while having no QAs might increase the pressure on developers. After all, testing your own code makes you look from the same perspective twice, which means you might miss some bugs. In other words, both situations have pros and cons. As there is nothing set in stone, it is up to teams to define this need.
See also: How we validated Flutter at scale for funda
Situation in funda's Search Team
In the Search Team, we are implementing the new search stack. Currently, we are running the search feature on ASP.NET MVC on a monorepo, which is hard to maintain, deploy and test. Therefore we are moving into a structure with microservices, Elastic and prebuilt UI components from appbase.io. At the same time, we are re-writing the mobile search with Flutter. The search is the core of the user experience: if users can’t search, they won’t be able to find any houses or schedule any viewings. And then there will be chaos!
In our team, there are:
- 1 product owner
- 2 backend developers
- 2 frontend developers
- 2 Flutter developers
- 1 UX designer
- 1 QA engineer
As you can see above, there is only one QA to test the output of six developers. Imagine getting 6 different user stories to test at the same time. Our QA will be:
- An octopus who tests everything at the same time,
- The bottleneck of the team who keeps 6 tickets in the ‘To Test’ column,
- An overlooker who just drags the tickets to the ‘Done’ column without properly testing them.
Or our QA can be a teacher/coach/mentor to other team members, showing them how to test the products.
Nowadays, we see a shift towards testing not only being the responsibility of the QA Engineers, but of the team as a whole. I will give you 5 reasons why this has many advantages:
1. Unblocking the stories in the pipeline
Imagine it is the second week of the sprint, and the sprint is about to end. If 6 developers put all the stories to the ‘Test’ column at the same time, it is impossible for a single QA to finish everything before the end of the sprint. So how do we tackle this?
Before we pick up more tasks, we focus on the existing task blocking the pipeline and test it together. More people means more pairs of eyes, more bugs found and higher quality.
2. Leads to better usage of the test pyramid
A test pyramid is a model to organize your tests into 3 major categories: unit, integration and E2E tests. In numerical terms, you have the unit tests most, then integration tests and then E2E tests are the least.
When testing as a team, we pay more attention to the test pyramid. The more the test pyramid approach is followed, the better the workload of writing tests is distributed. To give an easy example: if developers implement more unit tests, the end product will be less buggy, so there will be fewer regression issues, which leaves everyone more time to do exploratory testing or automate more tests.
Automated tests are the key to having stable, reliable and fast quality assurance. Our teams always make sure that everyone is capable of writing unit tests on backend/frontend. Having a feature unit tested inside builds/pipelines is a must in the acceptance criteria of our user stories.
Source: https://automationpanda.com/2018/08/01/the-testing-pyramid/
The following scenarios might occur based on the different usages of the test pyramid:
1. If developers implement no or few unit tests, QAs need to cover them with more E2E tests, which will end up in unnecessary, slow/flaky tests.
2. If developers implement a lot of unit tests, QAs need to implement fewer E2E tests, which will result in faster pipelines.
3. In an ideal world, the whole team implements many unit tests, some integration tests and few E2E tests.
Again, having one person responsible for one thing (such as a developer unit testing their own code or only 1 QA writing E2E tests) is not a safe approach. Double-checking everything with multiple pairs of eyes is the best approach.
3. Better quality
Testing as a team leads to better quality. When the same person (usually a QA) tests the same product or feature again and again, it leads to some kind of blindness. Repeating the same steps a hundred times causes a QA to lose the ability to look from the user's perspective. Many user experience issues, bugs and defects are overlooked by memorizing the clicking steps and just blindly following the same steps as you did in a bug ticket 2 days ago.
When a team tests a feature together, on the other hand, it brings a fresh perspective from new pairs of eyes. Many issues that are overlooked by developers and QAs can be detected by other developers, product owners and UX designers. As all the features are thought up by a product owner, designed by a UX designer, and implemented by a developer, everyone chips in with the product in one way or another. So, bringing their perspective and making sure that the end product looks good from everyone’s point of view is important. Finally, it makes everyone responsible for the end product.
4. Shared responsibility
We always try to deliver bug-free products, but unfortunately things can go wrong. When things do go wrong in software development – i.e. when a bug slips into the production – QAs usually get the same question: ‘how did you miss this (obvious) bug?’ However, quality issues are never one person’s fault; it is a fault in the whole process. Maybe the product owner wrote an incorrect description, maybe the developer implemented it the wrong way, maybe the UX designer designed an ‘impossible to implement’ sketch, or maybe the QA did sloppy testing. Instead of one person testing everything, testing everything as a team is beneficial to cross-check each other’s work and reduce the responsibility on the QA's shoulders.
When a bug slips into the production, our first aim is to hotfix or revert it as soon as possible. Then we think about having a proper fix. Finally, we organize a postmortem and discuss what went wrong. During these postmortems, we try to analyze what went wrong, how it went wrong and how we can take measures to prevent it happening in the future.
See also: Machine learning: the model behind funda's Waardecheck
We don’t always wait until something goes horribly wrong in production. We have the same process in the retrospective meetings. We don’t just chat for an hour and leave the meeting, but aim to draw up action points and follow up on these immediately. Non-actionable items mostly just disappear and are forgotten.
Although it is easy to say, there are a few ways to follow up on action points. What we try to do is:
- Avoid abstract outcomes
- Come up with concrete, practical targets
- Assign an action point to one person as a follow-upper, so it is not forgotten
- Have deadlines (unless it is a continuous habit)
5. No more: ‘Sorry, our QA is on holiday!’
This might be the biggest perk of testing as a team!
Most of you might have heard this in your work environments before: ‘I can’t run that test/process as only our QA knows how do it, but they are on holiday.’
Testing as a team eliminates the strong dependence on the QA engineer. Whenever the QA is on holiday, calls in sick, or takes a day off, any team member can just pick up the testing tasks.
As a QA, how can I convince my team members to test too?
If you have read up to here and also think testing as a team is a good idea, you might have these questions:
- ‘How do I convince other team members to test the product?’
- ‘If everyone tests the product, then what am I going to do as a QA?’
Here are 5 ways to get everyone to test:
1. Be vocal about the issues in your team
If you as a QA are the bottleneck of the team, be vocal and talk to your team members about this issue.
2. Do pair testing with your team members
If your team members don’t know how to test something, sit together or have a video call. Share your screen and go through it step by step, describing how you are testing something. What are the first things you look at? How do you set up the Flutter test environment and Android Studio?
3. Document your learnings
Instead of repeating the same process 6 times in a row, you can document what you know. Having a screen sharing session and installing Android Studio and Flutter might take too much time. Document this installation step, and share the guide with your team members. If they are stuck, help them to unblock and let them continue on their own.
4. Be the keeper and watcher of quality
Make sure that all team members are testing their own and each other’s code. If you see a quality drop or significant issues on deliveries, be vocal but be nice, and try to come up with suggestions and solutions.
Source: Marvel Studios & Paramount Pictures
5. Let your team members make mistakes
This is a controversial topic. If you are testing a medical device or anti-stall software for a commercial aeroplane, you don’t have the luxury of making a mistake. But if your software (or area) can tolerate mistakes, allow your team members to make them. Let them understand what went wrong and how they can make it better next time. If you do ‘helicopter parenting’ on your team members:
a. You will be exhausted as a QA,
b. Your team members will never learn from their mistakes, and
c. QA will be a safety net again and it won’t be a proper team effort.
Conclusion
At funda, we believe that testing as a team – instead of one QA being responsible for all testing – helps us to deliver better quality products.
Each team member has the same vision on quality. It reduces the workload of our team members – especially QAs. If things go wrong, it helps us to avoid blaming each other. We build everything together, so we test everything together.
