Aim for five to eight people on each backend team
* This article is a translation of the Japanese article written on December 25, 2021.
Hello. This is @godric, the Manager of Managers of the Merpay Backend Team.
This article is for day 25 of Merpay Advent Calendar 2021.
Merpay’s mission is "building trust for a seamless society," and our current profitability strategy is focused on our credit business. At the same time, we’re working on several new developments to seek new growth axes for the Mercari Group, such as Mercoin’s crypto-asset investment business and Souzoh’s Mercari Shops. These are crucial business strategies for Mercari Group, so we’ll need to maintain a high level of quality,launch and evolve services at high speed, and maintain our position in the market.
Although everyone on the Merpay Backend Team has been working in an All for One manner to support rapid growth for both Mercari and Merpay, accumulated technical and organizational debt has begun to present various issues.
In order to resolve the issues listed below, we began running an analysis of technical and organizational debt in the Merpay Backend Team starting in March 2021, and as a result began reorganizing some backend teams into groups of five to eight people.
- Domains managed by each team are inappropriate (too narrow or too wide)
- The number of people entrusted to each Engineering Manager (EM) is not optimal
- There is technical debt caused by other teams, making it much harder to resolve
You might think that team size isn’t a factor in all of these issues. However, by making adjustments with team size set as our goal, we can resolve other issues at the same time.
In this article, I’ll cover the topics listed below. I hope this will be of some use to others currently facing similar engineering team organization and optimization issues.
- Why five to eight people is a suitable size for a single backend team
- Steps for determining team reorganization
- Reorganization case studies
Why five to eight people is a suitable size for a single backend team
Team size isn’t the only factor involved in designing a team. Although there are a lot of factors that need to be considered, the goal is to maximize team performance by minimizing communication costs. We put people that communicate a lot on the same team, and those who communicate only sparingly on another team. This is similar to the system design principle of "Low Coupling, High Cohesion."
Merpay development team structure
Merpay uses the matrix organization structure shown in the graph. A single "domain team" is formed from people of various roles, including PMs, backend engineers, client engineers, and QA engineers. Backend EMs manage backend engineers on one or more domain teams. In this article, I use the term "backend team" to refer to the backend engineers on a single "domain team."
Conway’s Law
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.
— Melvin E. Conway
I think you’re probably familiar with Conway’s Law. For Merpay, we decided from the start to adopt a microservice architecture. Since then, we’ve kept Conway’s Law in mind when designing engineering organizations (especially backend organizations). We assign several specific microservices as well as development and operation responsibilities to each domain team.
The Inverse Conway Maneuver
Conway’s Law asserts that organizations are constrained to produce application designs which are copies of their communication structures. This often leads to unintended friction points. The ‘Inverse Conway Maneuver’ recommends evolving your team and organizational structure to promote your desired architecture. Ideally your technology architecture will display isomorphism with your business architecture.
https://www.thoughtworks.com/radar/techniques/inverse-conway-maneuver
Based on the name, you might think that the Inverse Conway Maneuver is simply the opposite of Conway’s Law. However, that is not the case. Conway’s Law states that, "any organization that designs a system produces a design whose structure is a copy of the organization’s communication structure." The Inverse Conway Maneuver means that, "If you design an organization based on the ideal system structure, you can naturally arrive at the ideal system structure."
When you need to adjust the assignment of domains between teams (in such a case, sometimes it would also require changes to system designs), designing or reorganizing the teams first can make it easier to arrive at the ideal domain assignment and system design.
I believe that team size is the criterion for determining whether the scope of the domain the team is now responsible for is appropriate.
Adjusting backend team size
I’ll limit my discussion here only to backend team size. There are three reasons for doing so.
- The number of people in each role except for backend engineers (PM, iOS/Android/frontend engineers, etc.) on a domain team doesn’t change much. In many teams, there is just one person per role.
- There are many engineers on a backend team, so a Backend EM will often work more closely with the domain team, compared with EMs of other roles.
- Microservices are developed and operated by backend teams, so the crucial adjustment points here would be the numbers and sizes of the microservices and domains assigned, and the number of team members in on-call rotation.
A backend team of four or less will have a high operation workload, and will be difficult to handle when multiple new projects come at the same time.
A backend team of four or fewer will not have enough people for on-call rotation. On average, each individual member will be very busy will on-call duties and will have trouble maintaining a healthy life rhythm. If a single member of the team goes on childcare leave or extended leave, the other members of the team will have much more on-call workload.
Furthermore, if the domain team is asked for the support of multiple projects at the same time, the too-small team could easily have resource conflicts among projects, preventing these projects from going in parallel.
And with a too-small backend team, the domain scope that can be handled is limited, making it more difficult for engineers to make a larger impact.
From an EM’s perspective, if there are few people to manage, an EM will either work as a playing manager, or will manage other teams simultaneously. If an EM is managing other teams, it will become difficult for them to focus on a single domain, increasing communication costs. Finally, members leaving or transferring will have a significant impact on a too-small team, making it difficult to manage risks.
A backend team of nine or more increases communication costs and reduces efficiency.
A backend team of nine or more might result in a domain team of 14 or more. Imagine what the morning stand-up meetings would be like for that team. Not very efficient!
In The Mythical Man-Month, Fred Brooks states that communication overhead will be O(N²) for a team of size N. In other words, communication costs will increase at a speed exponential to team size.
Larger teams tend to handle larger domains, which increases the amount of knowledge that each person needs to grasp in order to do on-call and development. The basic workload of the team is increased, thereby reducing the capacity of each person.
From an EM’s perspective, by taking too much time for people management because of the too-large team size, there is not enough time to invest in organizational issues, strategic planning, and other critical tasks.
The following table summarizes the impact of team size on team performance factors.
Steps for determining team reorganization
By firstly setting a team size of five to eight as our goal (while also taking the team’s actual situation into consideration), other parameters such as communication costs, domain, operation workload, and EM responsibilities can be tuned together. There are several teams within the Merpay backend organization with size or domain issues, so we reorganized these teams based on size.
Step 1. Create a domain list and migrate inappropriately assigned domains to the appropriate teams
I mentioned earlier the issue of the domain a particular team is managing not being of appropriate size. The assignment may not be appropriate for the following reasons.
- Due to team resource limitations at the time, the domain’s features were implemented by another team
- We were not properly discussing which team should manage the domain in the first place.
- Conditions changed and now it would be better for another team to manage that domain
If we can assign domains more appropriately, we can more efficiently discuss which team should be responsible for what during the planning phase of a new project. This can also reduce the number of teams involved in a single project, thus can reduce communication costs.
The first step for the Merpay Backend Team was to list domains managed by domain teams, and then to identify domain assignments that seemed inappropriate. We discussed the domain assignments with not only the responsible domain team but also other related teams, and determined whether to migrate the domain to another team (and if so, when).
Step 2. Reorganize teams into teams of five to eight, and optimize team sizes
We analyzed issues identified in Step 1 along with solutions to those issues and prepared proposals to transfer domain ownership. We then ran an analysis on the domains managed by backend teams to determine how many members were really required on each team. The estimations also showed teams of four or less, and teams of nine or more.
For teams of four or less, we proposed to combine teams who have domains with high affinity. For teams of nine or more, we considered separating the team.
Step 3. Work out team reorganization plans together with system migration plans, and resolve technical debt
In order to support the rapid growth of Merpay, there were cases where technical debt was allowed as a trade-off for business speed. In cases where microservice migration involving multiple teams is required in order to pay off this debt, we take this into consideration together with domain migration, and employ team reorganization that can repay the debt more easily.
The important thing here is to clarify the system and domain migration plan and its schedule together with the destination team. Schedules can change due to priorities, but working out a schedule together can still be important to realize the migration plan.
Reorganization case studies
Case study A: Approach taken for DX and UX issues with payment history
Solution: Transferred payment history domain to Payment Core Team
Previously, the logic for generating payment history was maintained by the team that manages an aggregation API called merpay-api. The logic behind this is quite complicated, and the team managing the aggregation API originally doesn’t have the responsibility to manage payment history. It was also very difficult to develop and maintain payment history using the aggregation API architecture. When new balance types or payment methods are added, or someone wants to further improve the history UX, the DX (developer experience) is poor and communication cost is high.
We consulted with the Payment Core Team, and decided to develop a new payment history service within the Payment Core Team (which should originally have maintained information related to payment history), and migrate the payment history domain to the Payment Core Team.
Result: The Payment Core Team is currently diligently developing the new history service. Because the domain has been assigned more appropriately, we can expect a decrease in communication costs as well as a significant UX improvement.
Case study B: Approach taken for issue ensuring sufficient on-call personnel on two teams involved in merchant domain
Solution: Combined two teams into one
Previously, the team in charge of the merchant platform was split into two domain teams (Operation Unit and Console Unit), with three and two backend engineers in each. However, each Unit was small and it was difficult to ensure sufficient on-call personnel.
If there were any projects, they were often developed together with the microservices they were each responsible for, so there was a lot of close collaboration between the two teams, and communication costs were considerable.
We therefore decided to combine these teams.
Result: More work can be handled by a single individual, and it’s now possible to consider integrated measures that cover the entire merchant platform. The combined team is better able to handle new projects, and although the team had originally considered to hire more people, they no longer need to do so.
Case study C: Approach taken for issue of inefficient communication within a team with many backend engineers (Payment Platform Team)
Solution: Split the Payment Platform Team into two teams: Payment Core and Balance
There were 12 backend engineers in the Payment Platform Team. The team had to split into two sub-teams to do sprint planning. When there was a joint meeting, due to the size of the team, it was difficult for individuals to voice their opinions.
We took this opportunity to completely split the team into two teams, and made it easier to assign EMs to the two teams. Although the team wanted to keep some joint meetings, we are now also considering holding these meetings separately.
Result: It’s now easier to make decisions on each team. The number of people participating in meetings dropped to half, making it easier for individuals to speak up.
In conclusion
There’s no single right answer when it comes to organizational design. Factors such as the company’s current phase, organization and individual growth paths until now, and system architectures will all have an impact. In this article, I covered our efforts to adjust factors impacting team performance together with team sizes. Having said that, there are still both small and large backend teams in Merpay. It will be important to always consider what is best for your current situation, and take Go Bold and Be a Pro approaches.
Reference material
Will Larson, An Elegant Puzzle: Systems of Engineering Management
Chris Richardson, Microservices Patterns: With examples in Java
Frederick P. Brooks Jr., Mythical Man-Month
