This is a translation of the original Japanese article.
Hello. I’m Tadashi, an Automation Engineer from Mercari’s Automation & Quality Assurance Group (AQA).
I mostly work on test automation and improving mobile CI /CD.
We tried HeadSpin’s service.
I would like to introduce the service and share some impressions.
What is HeadSpin?
HeadSpin is a service that provides device farm, test automation environment, and monitoring as 24/7 environment for monitoring the performance of mobile applications.
Using various networks, locations and devices, we can do performance test in a way that is close to what customers are using.
Motivation
We introduced UI test automation for both iOS and Android applications.
These tests are not connected to our QA environment; not to our production environment.
Therefore, current tests for production environment are now done manually.
There is a motivation to automate them and I thought that using HeadSpin would make it easy to test them in a way that is close to what customers are using.
In particular, use of:
- real devices
- networks such as 4G
- real locations such as Tokyo
In addition, with HeadSpin, we can also do performance monitoring in a production environment.
So we thought that we could also provide beneficial feedback to developers.
Features of HeadSpin
Device farm
In HeadSpin, real devices of various location and network are prepared as a device farm, and they can be controlled on Web UI.
Android device farm is based on open source tool called Open STF, and we can control it with the same operability.
HeadSpin is also compatible with iOS.
Screen operations, app installation, screen rotation, etc. are all possible from the Web UI.
Test automation environment
HeadSpin makes it easy to run automation tests on HeadSpin’s devices.
Test frameworks such as Appium / Espresso / XCTest / EarlGray are supported (see here for details)
In this trial, we used Appium for both iOS / Android.
With Appium, you can run tests by specifying the WebDriver URL and Capabilities displayed on HeadSpin UI.
# Sample code using Ruby def desired_caps { caps: { platformName: 'iOS', deviceName: 'iPhone', automationName: 'XCUITest', udid: ENV['HEADSPIN_UDID'], bundleId: ENV['BUNDLE_ID'], }, appium_lib: { server_url: ENV['HEADSPIN_URL'] } } end Capybara.register_driver(:appium) do |app| Appium::Capybara::Driver.new app, desired_caps end
Since Appium server is provided by the HeadSpin side, there is no need to install Appium on a local environment or a CI environment.
In this case, only Ruby execution environment is required, so we were able to easily create an testing environment using CircleCI.
version: 2.1 orbs: ruby-orbs: sue445/ruby-orbs@volatile jobs: test: docker: - image: circleci/ruby:2.5.3 steps: - checkout - ruby-orbs/bundle-install - run: bundle exec rspec workflows: version: 2.1 hourly_test: triggers: - schedule: cron: "0 * * * *" filters: branches: only: - master jobs: - test
Also, with Appium’s GUI tool called Appium Desktop, you can easily connect with HeadSpin’s devices.
Performance monitoring
The performance of devices during automation tests can be recorded as a Session.
In each Session UI, you can see the performance of devices such as CPU rate, frame rate, etc.
You can also see the response time from devices to the API, CDN, etc.
In addition, this data can be stored in a DB provided by HeadSpin.
You can continuously monitor it by linking this DB with BI tools such as Grafana or Looker.
Impression
Here is my impression based on this trial.
There is no need for an additional SDK
When using HeadSpin, additional SDKs are not needed in the app.
We did not have to worry about unexpected troubles and performance deterioration by putting them, which made us easier to introduce HeadSpin.
Uniqueness as iOS device farm, test automation environment, and performance monitoring
As mentioned earlier, OpenSTF supports Android but iOS is not supported.
HeadSpin is not only an iOS device farm, but it also provides continuous test automation environment / performance monitoring tools.
We can rely on it as a test automation environment to some extent
As mentioned earlier, because we can leave Appium and other test automation environments to HeadSpin, I felt that there are quite a few parts that we have to maintain.
In addition, adb · idevice commands can also be called via HeadSpin’s API to operate the devices, so it adds a lot of flexibility.
What monitoring and feedback we can give to engineers
Initially I was supposed to give feedback on performance for mobile engineers (iOS / Android).
This creates some complication because HeadSpin measures the performance that includes outside of the app such as network and etc.
It does not provide feedback at the code level of the mobile application.
If you want to provide feedback only for mobile applications, it will be necessary to combine it with another tool such as Nimble (also from HeadSpin).
Conversely, HeadSpin can measure the total performance including the response time to API and CDN in the same environment that customers actually use.
This may provide valuable information and feedback not only to mobile engineers but also to SRE and backend engineers.
Endnote
Mercari JP is growing in scale, so we believe that great user experience by various customers are becoming more important.
Within that, it is important to know what affects customer experience in the real world environment.
Although it will not have an immediate effect, I think we need to take action to improve customer experience based on the information / feedback obtained from HeadSpin’s service.
