Note: This is a translation from the Japanese article.
Hello. I’m Tadashi, an Automation Engineer from Mercari’s Automation & Quality Assurance Group (AQA).
I mostly work on test automation and improving mobile CI/CD.
I’ll talk about a test automation environment which can run Appium/Android tests with 20~30 emulators.
-
Current Android test automation environment and its problems - Docker-Android
- How to create in the cloud
- Benefits of using Docker-Android
- Problems to be solved
- Endnote
Current Android test automation environment and its problems
We use Appium for Android UITest automation.
When we first created tests, they were run every time before release as a regression test.
At that time, the test automation environment was the following:
Jenkins (master) -> mac (node) -> Android real device
It might be enough for a test automation environment at first.
Currently, we are increasing both test cases, test automation engineers, and the frequency of running tests.
We faced some problems due to this situation.
- Increasing execution time
- Maintenance cost of real devices
- Test failures because of device condition such as network
- Maintenance cost of local environments such as Appium, Ruby
We use Swift / XCUITest / CircleCI(iOS Emulator) to run tests stable, so I was thinking of running these Appium/Android tests in the cloud.
It has become common to run Android UITest in the Firebase Test Lab.
However, Firebase Test Lab currently does not support Appium, so we tried a tool called Docker-Android.
Docker-Android
One of the tools that I used to create this test automation environment is called Docker-Android, a Docker image.
This tool was introduced in AppiumConf2018 in London.
This Docker image has the following features.
- You can use the x86 Android emulator, which is better in performance than the ARM Android emulator.
- You can start the Appium server inside the container using environmental variable
APPIUM=true. - You can connect this container to Selenium Grid easily.
- You can see the behavior of the Android emulator inside the container by using noVNC.
- Can capture videos.
You can create the Appium + Selenium Grid test automation environment by writing docker-compose.yml below, and running the command$ docker-compose up -d
# docker-compose.yml version: "2.2" services: selenium_hub: image: selenium/hub ports: - 4444:4444 nexus_5_8.0: image: butomo1989/docker-android-x86-8.0 privileged: true depends_on: - selenium_hub environment: - DEVICE=Nexus 5 - CONNECT_TO_GRID=true - APPIUM=true - SELENIUM_HOST=selenium_hub
You can also scale emulators by adding options --scale nexus_5_8.0=XX if the same Node.
How to create in the cloud
We wanted to create this test automation environment in the cloud, considering maintenance cost and stability.
I expected to be able to easily create it because it’s provided by the Docker image.
However, there are restrictions for using this Docker-Android in the cloud.
Docker-Android needs hardware virtualization because it uses x86 Android emulators.
We cannot use this feature in general cloud instances because they are run under hardware virtualization.
Therefore, we needed special configurations to run Docker-Android.
Enable Nested Virtualization
You can use hardware virtualization features inside the virtual machine.
You can use this feature in Google Cloud and Microsoft Azure.
Enabling Nested Virtualization for VM Instances | Compute Engine Documentation | Google Cloud
Nested Virtualization in Azure | Blog | Microsoft Azure
Using Bare Metal Instances
Currently, AWS provides Bare Metal Instances called i3.metal
Amazon EC2 Bare Metal Instances with Direct Access to Hardware | AWS News Blog
There are documents or sample codes to use Docker-Android in each cloud services.
docker-android/README_CLOUD.md at master · budtmo/docker-android · GitHub
We used the AWS Bare Metal Server referencing this article.
i3.metal is much better in performance compared to other instances so we were able to create a test automation environment with multiple Android emulators.
Benefits of using Docker-Android
Finally, we created a test automation environment which can run Appium/Android tests with multiple emulators.
We expect to decrease test execution time with this.
Currently, we run these tests before release and every night, but we expect to run more often this environment.
We also expect decreased maintenance costs by using emulators, Docker containers, and the cloud.
Problems to be solved
We got the above benefits, but we still have problems to be solved.
Divide roles between emulators and real devices
As I said, this test automation environment uses Android emulators.
However, it’s difficult to cover all tests with emulators because some tests like camera tests need to be tested with real devices.
So we need to divide roles between emulators and real devices.
Server load to QA environment
Currently, this environment is separated from the QA environments.
Therefore, the QA environment will receive high server load when the number of parallel execution increases.
Some actions will be needed such as using mock servers or scaling the QA environment automatically.
Distribution of tests and test reports
Currently, we divide tests manually with Jenkins pipelines for each report.
We’ll need to consider how to divide tests because mobile UITests takes time.
It’s better to use some tools such as Knapsack to optimize distributed tests.
Also, we need to update test report functionality for distributed tests.
In SeleniumConf Chicago, there was an example using Allure for distributed selenium tests.
I think it can also be used for our tests.
Allure | Test report and framework for writing self-documented tests
Endnote
As I said above, we still have problems to be solved.
In addition, we need to continue working with developers and QA on what to cover in UITest.
However, I believe faster is better in test automation, and I believe decreasing execution time will make this test more valuable.
We’ll continue to strive for a better test automation environment.
