The eleventh day’s post of Mercari Advent Calendar 2020 is brought to you by Ken Wakasa (@kwakasa), VP of Product Engineering, from the Mercari Engineering division.
Let me take this opportunity to recap the challenges we have been facing to assure the quality of our products as well as some of the directions and actions we have been taking to achieve our goals.
Large-Scale Growing Products
At Mercari Engineering, we are committed to building products and services that deliver value to our customers. This year too, we’ve made tons of new feature launches and improvements to get closer to our mission. While we are very glad to see that we serve and delight more and more customers, our engineering team keeps adding new code and making changes to our huge codebase.
As a matter of course, whenever we make changes to the source code, the product’s behavior changes. We also know that the bigger and more complicated the codebase and functionalities get, the more likely we encounter unexpected behaviors or bugs with any changes. It’s actually not very straightforward to maintain and improve the quality of large-scale growing products.
Quality of Products
To clarify what we are aiming for, let me summarize some aspects of the product quality we care about.
Implementation
In my view, there are two subcategories of the implementation quality.
External Quality
It includes several aspects like adherence to specification, functional suitability, performance efficiency, no UI glitches, no crashes, and so on. Failing to meet them would directly impact the customers and be externally visible.
Internal Quality
There is another area of the implementation quality that is not visible externally. It covers aspects such as code readability, code maintainability, code testability, and so on.
Specification/Design
Another important aspect of product quality belongs to specification and design. They have impacts on customer experience, usability, functionality, accessibility, security, etc.
Responsibility
We believe engineering teams should be responsible for everything related to the implementation quality because engineers are obviously the ones who write the code and understand the implementation details the most.
It should be pointed out that the internal quality is actually one of the most essential elements to achieving the external quality of the implementation. That’s why people emphasize the importance of code quality, code readability as well as unit tests to increase code testability and maintainability. Test-driven development (TDD) with unit tests is now a standard that many engineering teams follow. This post does not talk much about it because we have many good resources on it out there.
On the other hand, which team(s) should be responsible for the specification/design quality? It ultimately should be all teams. Considering that engineers are ultimately the experts of the implementation ("how we build"), I’d say the quality of specification/design ("what we build") should be owned primarily by QA and Product teams while engineers definitely should work closely with them. We have many good references for these topics out there as well.
This post rather focuses on what engineering teams can do to detect and address external quality issues in the implementation.
Challenges
One of our biggest challenges was that we cannot afford to have enough testers as our product grows with many functionalities. It was becoming clear that we would not be able to rely only on testers to assure the quality of our products.
Another challenge was that it was unclear who is responsible for quality. There was no clear common understanding of it — some engineers might say it’s the QA team’s responsibility to find out any quality issues, QA team might assume it’s the engineering team’s responsibility to write quality software, and so on.
You may say "why don’t we hire more testers". Yes, it might work, but it’s easier said than done. We all know it’s extremely difficult to find enough skillful testers who can run manual tests against ever-growing features and functionalities. Mercari Engineering made a decision not to take that path. While we recognize the importance of manual exploratory testing which we will keep conducting, we wanted to invest more to make our engineering practice and process scalable and sustainable to achieve quality. To make our message clear, we stopped adding more testers to the teams.
It was actually a bold move from the engineering team’s perspective. Engineers in some teams cannot help but conduct manual testing until the tests get automated. We started hearing engineers’ feedback like why engineers need to do QA, testing is boring, engineers are demotivated, and so on. Wait a minute, why does it make sense to ask someone else to do a "boring" task instead of you? Why don’t we rather think about making QA more creative and fun? It was obvious that we had to go ahead and change engineers’ mindset on quality assurance.
What Do Software Engineers Do?
Engineers ≠ Programmers
Writing code is part of the engineering work. Engineers are expected to be very good at it, but it’s not a sufficient condition for being a great engineer.
While there are various opinions on the definitions of "programmer", let me try distinguishing software engineers from programmers from my point of view.
Programmers
Programmers (or coders) write code. They are responsible for Development.
Software Engineers
Software engineers have a broader scope of job responsibilities to build viable and quality products. They are responsible for Design, Development, Testing, Maintenance, Evaluation, and so on.
We would like our engineers to do engineering, not just programming or coding, by adopting whatever engineering measures take to build quality products. We want to get rid of the "we write, they test" mindset from Mercari Engineering.
Engineering Excellence
In our engineering excellence vision, we are aiming to "get rid of dependencies on specific engineers" for our scalability and sustainability. It goes for the QA team as well. We are trying to stop depending on a specific team such as the QA team to assure quality.
"Automate everything" is another aspect of engineering excellence. Engineers know the values and power of automation. Why don’t we do the same for quality assurance? Whenever we find something that can/should be automated, let’s do it. We assume that, by conducting manual testing even reluctantly, engineers can easily notice what to automate.
Vision of "Software Engineer"
Mercari Engineering is advocating a vision dubbed as "Everyone is a Software Engineer" which means that we are not backend engineers, machine learning engineers, frontend engineers, nor mobile engineers. We all are software engineers who make use of expertise in specific technology domains and even explore unfamiliar domains when needed to accomplish our goals as a team.
This is of course applicable to QA too. All engineers are expected to take ownership of the quality of what they build. Let’s stop focusing on the roles and start focusing on the products.
Directions
Quality Responsibilities of Scrum teams
We’ve been adopting Scrum for our development process. We created a guideline that makes it clear that the Scrum team owns quality, not a person in the team. We believe a team can build quality products when the entire team is responsible for quality.
All members (Engineers and PMs) of each team is now responsible for:
- Reviewing and defining spec and test scenarios
- Creating, reviewing, and refining test cases with the QA team
- Confirming release timelines contains time for automated and manual tests
- Maximizing the ratio of automated integration/end-to-end test cases as well as unit tests
- Minimizing the ratio of manual cases
- Conducting dogfooding and manual exploratory testing as needed
- Getting a QA team’s sign-off for automated and manual test coverage
- Assessing the QA situation and make corrections
Team members are expected to iterate and optimize the development process to assure quality.
Engineering-driven QA
One of our goals is to assure quality by engineering as much as possible.
The following ice-cream cone describes what many teams, including us, often end up with, unfortunately.
The following pyramid describes the QA foundation we want to build our products upon.
As mentioned above in the Engineering Excellence section, we should apply "automate everything" wherever we can. It’s the engineering-driven QA we are striving to get to.
Production Code and Test Code
We really don’t need to distinguish the test code from the production code. We should consider writing and maintaining test code is actually part of product development. In other words, QA should be embedded in the codebase to achieve quality.
QA ≠ Test
I feel we somehow tend to forget that QA is an acronym of Quality Assurance that is never the same as testing nor manual testing. Quality is not equal to test. Testing is part of QA and not the only element of QA.
QA Team’s Responsibility
As we defined the Scrum team’s responsibilities on quality, the role of the QA team started shifting as well. The QA team’s new responsibilities are:
- Building a comprehensive QA process
- Supporting Scrum teams in managing test cases and executing QA
- Maintaining a knowledge base of QA
- Accelerating integration/end-to-end test automation with Scrum teams
- Focusing on exploratory tests from a customer perspective
- Signing off for release
- Implementing a data-driven improvement process for customer experience
Engineering teams are encouraged to get help from the QA team as needed, but it does not mean they can rely on the QA team to execute QA. We’d rather like to have the QA team focus more on the quality of product spec and customer experience.
Feature Flags
Our engineering teams are making use of Feature Flags intensively to achieve data-driven development. Primary use cases are phased rollout of features and A/B tests. It’s also actually one of the use cases to use Feature Flags as a kill switch if something goes wrong in production.
From the quality perspective, it’s a recommended practice to always guard a new feature with a flag. No matter how careful we assure quality, we unfortunately sometimes hit production issues. A kill switch can save us as a last resort.
Company-wide Dogfooding
Rather than conducting manual regression testing by a dedicated team, it makes a lot more sense to have an organization use its own pre-release product in order to find bugs and receive real-world product feedback.
Our engineering team has been working on setting up the process and infrastructure to distribute pre-release versions to internal dogfooders. It’s actually helping us not only catch issues early on but also motivate the team to focus on quality.
Concluding Remarks
At Mercari Engineering, we are advocating the vision that engineers are responsible for the quality of the implementation. We think engineering is much more than just about writing code. QA is part of engineering. It should be embedded in development. We are aiming to apply engineering excellence to our QA effort.
To achieve our vision, we have identified challenges we were facing and come up with our directions on quality responsibility and recommended engineering practices. We recognize test automation is a must-have, not nice-to-have to build quality products in a sustainable way.
We are still on our way to get there. We are committed to having quality embedded in our engineering culture!
Mercari is looking for Software Engineers who share our mission and values.
https://careers.mercari.com/job-categories/engineering/
Tomorrow’s blog post —the twelfth in the Advent Calendar— will be written by @kinnerapriyap. Hope you are looking forward to it!
