Hi, I’m @celia, an iOS Engineer working at Merpay. Last time, I wrote about the different aspects of infrastructure in Mercari’s mobile team, and introduced a few tools we made that makes our team more productive (article is available here). Today I want to share with you about some improvements we made recently, on automating code integration between Merpay and Mercari iOS.
Background
Let me first explain the problem we are trying to solve. As you may have guessed (because we have a code integration problem), Merpay and Mercari’s iOS codebase is stored in different GitHub repositories. Merpay’s code exists in Mercari app as a git submodule, and it is managed using a dependency manager called Carthage. Other than develop, commit and push, updating Merpay code in Mercari is an integral part of our tasks.
If you are an iOS developer who is familiar with Carthage, feel free to skip this part. For those who don’t, let me explain how Carthage update works. It uses a file named Cartfile to keep track of dependencies for your project, pointing to the corresponding version or git position that it’s checking out from. In Cartfile, the dependency information is specified in the following format:
github "owner/merpay-ios-sdk" "1.3.0"
To update it, overwrite the Cartfile to the latest position, and run command carthage update in the console under your project directory. You can find details in Carthage’s documentation.
The Mercari iOS app is usually released on a bi-weekly basis. Before, we assign one member from Merpay iOS team to handle all the tasks, mainly:
- Confirm all pull requests under this release target is merged
- Update version number and add git tag for it
- Merge into main branch
- Update Carthage dependency in Mercari
We had a detailed checklist stored in GitHub’s issue template, and for every release we open a new issue to manage it. It was a series of tasks that took a few hours to do, and if something went wrong, might take half a day of our member’s time. This means the member we assigned to work on it needs to stop their tasks at hand, and deal with integration on high priority.
To automate this process, the rough requirements of our automated solution are:
- Increment version
- Create pull requests and merge branches
- Add git tag
- Make request to Mercari CI to trigger a job to update Merpay
- Overwrite Cartfile and run
carthage update, submit a pull request in Mercari GitHub repository
We also found time was mostly spent on waiting for CI to run tests, which happens when a branch is merged or a new commit is pushed. Since the only relevant code change was bumping up the version number, which has no effect on the test results, it should skip all CI checks to speed up the entire process.
Solution
We set our goal to shorten the integration time to be within 40 minutes, and automate all the manual steps. We also consider this an opportunity to build a tool that’s versatile, so we can use it to build more automation features in the future. In addition, it should be easy for anyone to maintain and debug.
After discussing the idea with people across different teams, we figured out we can build a Slackbot using Google App Engine, that it parses simple incoming arguments and makes HTTP requests to CircleCI pipeline to trigger a workflow, which runs custom scripts to perform different steps in the integration tasks. We can write scripts in Ruby using fastlane to achieve steps that can be done using shell scripts and GitHub API, as long as we are able to connect the workflow so it starts next task in the sequence when the previous one is finished, and we are able to make a request from Merpay’s CI to Mercari’s CI to trigger a job to update Merpay, we can achieve our goal. Previously, we already solved the problem of overwriting Cartfile and running carthage update using scripts, you can find more about it in my previous blog post.
Here is a flow chart representing each step in the automated integration process:
A bot command prepare_next is triggered on Slack. It sends a message to Google App Engine (GAE). Then GAE parses the message and makes a request with needed parameters to CircleCI using pipeline. CircleCI then runs custom scripts written in fastlane, that it checks out the latest Merpay code, bumps up version number, adds tag for it, and sends another request to Mercari’s CircleCI using pipeline to trigger a job to run scripts to update Merpay.
While the job is running, it will post progress to the Slack channel mentioning Merpay iOS team, and when the workflow is finished, it will generate a pull request, updating Merpay to the latest version. The integration is finished when this pull request is merged.
Example of Slack notifications, certain information has been masked due to security concern.
Result
With the help of this tool, we are able to decrease the time needed for integration in release from 168 minutes to only 12 minutes, which is more than 10 times faster. We are also able to remove all the manual steps so the workflow is triggered by one single bot command.
The significant decrease of time is made possible by skipping as many CI checks as we can safely, and avoiding the cost of engineers frequently switching between tasks and the need to work in both project directories.
At Mercari/Merpay, we are always looking for better ways to deliver our product safer, and faster. Automation not only reduces repetitive manual work and lets engineers focus more on actual product development, it also reduces risks brought by possible human error. In the future, we’d like to add more features to the bot, such as deploy beta apps or triggering tests running on CI. It has the potential to simplify other aspects of our daily work as iOS engineers, and I’m excited to see how it will evolve in the future.
Conclusion
At the end, I want to thank the following people for contributing to this project:
- @vvakame (advisor)
- @kenmaz (bot engineer)
- @celia (CI engineer)
- @takeshi.sato (bot name suggestion)
- @kaito.watanabe, @daichiro (code reviewer)
I hope this can inspire you to find areas around your daily work that can be improved by automation, and try to solve it with modern tools such as GitHub actions, CI, webhook and API provided by different platforms. It’s a great way to improve velocity of your software project, and a chance to try technology that otherwise you won’t come across.
