CI/CD (continuous integration, continuous delivery) is crucial for developing software and delivering value to users quickly. CI/CD is required at each technology layer at both the backend and frontend. In this article, we spotlight the Client CI/CD team, which provides CI/CD for the mobile app and web versions of Mercari.
We interviewed @y-kazama, @kaito, @thi, and @aha-oretama from the Client CI/CD team. We asked them about their current work, future plans, and the unique efforts the team has undertaken to date. Read on for their thoughts on how to ensure that engineers stay productive as an organization grows.
This interview was conducted by @afroscript who works for the Engineering Office.
Members of the Client CI/CD team
— Let’s start with some introductions.
@y-kazama: I joined Mercari in July 2019. I previously worked at a manufacturing company, where I was in charge of collaboration with an overseas company on developing enterprise-level document management software. Since joining Mercari, I’ve been managing multiple teams as an Engineering Manager (EM). I’m now responsible for team management, which allows me to help improve the performance of our team members.
@kaito: I joined the company in October 2018, right when a lot of new graduates were being hired from overseas, and I remember taking part in the hiring ceremony even though I wasn’t hired straight from university. I previously worked as an iOS engineer in charge of a recipe video application. I was involved in improving iOS app CI/CD back then, too. Prior to that, I had started a company when I was still in university, and as the CTO I was responsible for everything development related.
At Mercari, I’m involved in improving and automating iOS app CI/CD, collecting development metrics, and visualizing data. Although I’ve switched from one team to another, I’ve worked in the same area the whole time.
@thi: I joined the company in May 2021. Before that, I worked as an iOS engineer and was responsible for the build infrastructure for iOS apps. I was also involved in SDK development, in addition to user-focused development.
At Mercari, I’m mainly responsible for improving the build infrastructure for iOS apps. I maintain and improve the build system, which serves as the foundation of the build infrastructure.
@aha-oretama: I joined the company in August 2018. Prior to that, I was involved in large-scale backend and frontend development. I started doing Software Engineer in Test (SET) work on my own, and then shortly after that joined Mercari, which has a CI/CD team.
At Mercari, I’m responsible for CI/CD for the web version of Mercari and test automation in the QA Automation team and my current team. If it involves improving quality or productivity, I’m involved.
Metrics for Measuring CI/CD Productivity
— What does the Client CI/CD team do?
@y-kazama: The Mercari engineering organization is split into Product Engineering and Platform Engineering. Product Engineering is responsible for developing new Mercari application features. Platform Engineering is in charge of platform development to increase development productivity and system reliability. Our team is under Platform Engineering. We develop tools and infrastructure that support the CI/CD environment required for client-side development, such as for the mobile app and web versions of Mercari.
@kaito and @thi are responsible for CI/CD for the mobile app, while @aha-oretama is in charge of CI/CD for the web version. @y-kazama oversees the team as the EM.
— What was the CI/CD environment like before the team was formed?
@y-kazama: Originally, product development engineers were responsible for providing the CI/CD environment. However, CI/CD maintenance tended to get put off for later whenever they got busy with development. Then they would get loaded down with daily improvement tasks, and it became difficult to do CI/CD work from a medium-to-long-term perspective.
The Microservices Platform team does this kind of work on the backend, but we didn’t have a team that supported the mobile app or web versions of Mercari. That’s why the Client CI/CD team was formed in the middle of 2020. A highly productive CI/CD environment allows product engineers to focus on developing new features. This enables us to provide the Mercari application to users with greater reliability and speed.
— Is there anything you’ve done to keep your team running smoothly?
@kaito: Our team’s work is somewhat exclusive, and so we worry that it might be difficult to notice bottlenecks during product development. That’s why we’ve built an environment for detecting build times and other data.
@aha-oretama: We join product development projects and participate in regular meetings. This keeps us informed about what’s being developed and allows us to receive feedback on how we can help as CI/CD professionals.
We also do mob programming with product development engineers, which is a good opportunity to foster communication.
— How do you prioritize feature requests from product engineers with the tasks your team wants to accomplish?
@kaito: We prioritize requests by considering what would bring the greatest benefit or have the greatest effect on the development experience.
@y-kazama: Although we make minor improvements every day, there are also some things we want to do as a team over the medium-to-long-term. We make sure to maintain a good balance between both kinds of work.
— What are some things you keep in mind in doing CI/CD work?
@y-kazama: Mercari has a culture of applying data-driven decisions toward improving daily activities based on the data we collect every day. Similarly, our team is always thinking about what needs to be done to find and improve issues that might be harming CI/CD productivity. For example, we might introduce a system to increase build speed if building is slow for the mobile app, or provide a system to stabilize testing if integration testing is unstable.
— How do you determine CI/CD productivity?
@y-kazama: First, we define metrics for determining productivity. Next, we visualize these metrics on a dashboard. We focus particularly on build speed and stability, the developer experience, and then cost. If any of these metrics fall below our target values, we check logs and other sources to investigate why. Then we make improvements and monitor the situation to see whether the metrics actually improve.
For the developer experience metric, we have product engineers fill out questionnaires so that we can gather qualitative data.
— What are some difficulties you’ve had?
@kaito: The frontend overhaul project was pretty difficult. We had decided to introduce a build tool called Bazel and were building a CI/CD environment, but there weren’t many examples of using Bazel that we could use for reference, which made it hard.
The build method and tools used for Bazel are different from those typically used for iOS development. This made it difficult to resolve dependencies, and because we couldn’t use anything provided by the iOS ecosystem, we had a hard time integrating Bazel.
@thi: As Bazel is not the default build system for iOS, it is not being used by most general iOS projects out there, so there aren’t many iOS engineers that are used to using it. It was difficult to maintain a good balance when introducing Bazel. For example, we wanted to make sure that it would be easy to add dependency libraries and start development, even for someone who had never used Bazel before.
@aha-oretama: For the web version of Mercari, we used Web Components when overhauling the frontend. Web Components is a new technology, so we ran into situations where it wouldn’t work with certain libraries or where there just wasn’t any information out there about what we were trying to do. This made development and testing more difficult. Web Components didn’t work well with our existing testing library, so we were worried about whether developers would actually be able to easily write tests. However, we ended up building an environment that made it easy to write tests because we had supported this from the very start of the project.
— What is the most important point to consider for improving CI/CD?
@y-kazama: You need to ensure that product engineers can focus on developing products. It’s important to provide an environment that can be used reliably and on demand, kind of like the water or gas utilities we use at home.
@thi: There are four developer metrics: build speed, stability, developer experience, and cost. I mainly focus on build speed. Build speed and productivity can drop as the number of developers increases in proportion to the size of projects. In cases like this, we monitor the situation and make improvements, to keep any drops in build speed or production from happening.
@kaito: This is a continuous process, so high reliability is crucial. Here’s something that often happens with small teams. The team will introduce CI/CD but will often fail when it comes to testing, which means that CI/CD itself becomes unreliable. That’s why it’s important to successfully introduce CI/CD and then continue to maintain it.
@aha-oretama: CI/CD isn’t something you introduce and then forget about. It creates value only if you continue to use it. You often see it get reduced to a mere formality, especially if testing continues to fail. It’s important to keep providing reliable testing.
Technologies Supporting Mercari Client CI/CD
— What are some of the technologies and tools your team uses a lot?
@aha-oretama: For web development, we select whatever technologies and tools are ideal for the product requirements we’re given. I hope we’ll be able to standardize our tools as we hire more people and become involved in multiple projects.
@kaito: There aren’t many examples out there of using Bazel and Remote Build Execution (RBE) for the mobile app, so I think these will end up becoming unique tools for us. The build time tends to be the bottleneck for iOS development, so the tools interest me in that they can resolve this.
@thi: Bazel was originally developed as a build system for large-scale monorepos. It was designed as a scalable tool that would not impact productivity, even if the scale increased.
— How effective has Bazel actually been?
@thi: Let’s consider an example. A new engineer checks out the latest source code and builds it to begin development. This process takes around 30 seconds on average. If we weren’t using Bazel, the same process would probably take around 200 to 300 seconds; that’s seven to ten times faster.
@y-kazama: The source code is massive for a large application like Mercari, so it’s difficult to maintain good build speeds. Introducing Bazel has had a big impact in terms of further improving build speeds.
Building a Platform That Will Be Used by Even More Engineers
— Let’s look to the future. What are some technologies you’re looking forward to?
@y-kazama: I want to provide crucial infrastructures that engineers will be happy to use. We currently use a lot of third-party software and OSS to develop and run our tools. We’d like to be able to develop tools that can be used even outside the company in the future. We need more people if we’re going to achieve this, so we’re focusing on hiring now.
@kaito: We continue to overhaul the frontend for the mobile app and web versions of Mercari, but continue to meet technical challenges for each platform. I’m looking forward to seeing how much we can improve the development experience. I think the interesting thing about being on this team is that you get to work on the entire process of making development infrastructure improvements, from gathering data to discovering issues, and investigating and implementing solutions.
I’m also interested in seeing further improvements to Bazel. It was quite a technical challenge to introduce it, but things have improved a lot since @thi joined us. The company plans to hire more engineers, and so the amount of code will also increase. I’m looking forward to seeing where bottlenecks occur and how they can be improved.
@thi: I’m looking forward to providing a build platform that any of our internal engineers can use. The build infrastructure is the foundation of the software engineering process. It offers a lot of possibilities, such as ensuring that the productivity of our engineers doesn’t drop and allowing people to work from anywhere, even if they are involved in large projects. I’m looking forward to seeing what happens.
@aha-oretama: I want to build a platform that will be used by even more engineers. We’re a small team now, so we can only support a small number of projects. I believe that we’ll be able to extend our capacity by building the type of platform that engineers need.
— You mentioned that your team is small. Are there any specific projects or topics you haven’t been able to tackle, but that you’d like to take on in the future?
@y-kazama: The most recent thing that comes to mind is that there aren’t enough Android engineers, so our Android projects tend to get put off. If any Android engineers interested in CI/CD are reading this, please come work with us!
@aha-oretama: I want to develop features that can be used across the entire Mercari web environment. For example, visualizing building and testing across projects. This would allow us to extend the capacity of our team.
@thi: Right now our work only involves client applications, but going forward, I’d like to see our team take this system and apply it to the backend as well.
— Some of the people reading this interview might be interested in working for Mercari. What would you like to tell them about why you recommend working for Mercari and the Client CI/CD team?
@y-kazama: You can work on whatever you want, without any constraints. For example, @thi proposed introducing Remote Build Execution (RBE) and is now working on that. I also like that you can devote yourself 100% to infrastructure work, rather than just doing it when you have some spare time from product development.
@kaito: I like that you can get involved in the entire process, from discovering issues to investigating and implementing solutions. There aren’t many teams out there that mainly work on platform improvement on the client side. The number of engineers at Mercari is increasing, so the impact of making platform improvements is also increasing.
The scale of services is growing throughout Mercari as a whole, and in this environment we can try new technologies and make improvements. I also like how systems are updated regularly to make work easier, in order to further increase performance.
@thi: I like that there are plenty of interesting issues to solve in areas like build systems, toolchains, and build performance. If you’re interested in build-related work, I think you’d like working on our team.
@aha-oretama: I like that we can gather data and think about issues on our own, and then solve those issues. I’d definitely recommend this kind of work to anyone interested in working independently on the entire process from setting to solving issues, in order to improve development productivity.
Hiring Information
The Client CI/CD team is hiring! If you are interested in joining us, please take a look at the application guidelines on Mercari Careers.
