Note: This is a translation from the Japanese article.

tech.mercari.com

Hello. I’m Tadashi, QA Automation Engineer from Mercari.

Currently, I am working on UI test automation for Mercari’s web version.

This time, I would like to talk about UI test automation (Selenium WebDriver) with Microsoft Edge browser, and the testing environment for it.

• 
  UI testing strategy for multi-browser

• Running Selenium WebDriver tests with Microsoft Edge browser
• Challenges in the test environment
• Create a test environment on Azure DevTest Labs
• Cost comparison with other services
• Conclusion

UI testing strategy for multi-browser

Previously, I created UI test automation environment using Azure Kubernetes Service / Selenium Grid (I will write about this in another article).

For UI tests, creating the testing environment may be more complicated and it takes more time for execution, compared to tests at other layers.

We solved the above problem by running an appropriate size of UI tests on Azure Kubernetes Service / Selenium Grid.

However, because docker-selenium’s docker image is used in the above environment, tests could only be run on Chrome or Firefox browsers.

As you can see from the chart below which shows the web browser market share, there are many people who use Safari / Microsoft Edge browser.

It was clear then that we had to either run manual tests or find other means to run automated tests on these browsers in order for our customers to use Mercari’s web version comfortably

Up until this point we had implemented and UI tests using WebdriverIO, as Safari and Microsoft Edge browser had support for it.

blogs.windows.com

webkit.org

However, there is a limit to how far you can go trying to run all the tests on all of the platforms all the time.

The main reason for this is that Safari / Microsoft Edge both have a few restrictions on supported OS / environment.

• The desktop version of Safari only works with macOS, and macOS can only be run on Mac’s hardware under Apple’s software license.

• The desktop version of Microsoft Edge (currently available EdgeHTML version) can　only run on Windows 10 and requires a software licensing agreement / purchase.

These restrictions made it difficult to create and maintain Safari / Microsoft Edge test environments compared to the Chrome / Firefox browser environment.

For this reason, we realized that it was necessary to prioritize the execution frequency and the number of test cases to be executed by each browser, as shown in the pyramid diagram below.

Running Selenium WebDriver tests with Microsoft Edge browser

Microsoft WebDriver is required to run Selenium WebDriver tests with Microsoft Edge.

You can download Microsoft WebDriver from the following page.

WebDriver – Microsoft Edge Development

Please be aware that the installation method for Microsoft WebDriver is different for EdgeHTML between versions 18 and above and below 18. The Chromium version is still in preview stage.

Next, install Selenium WebDriver to Windows 10.

I recommend you install it via Chocolatey Windows package manager.

GitHub – dhoer/choco-selenium: Installs and configures selenium standalone, hub, or node server

To register as a Selenium Grid Node on the hub, execute the following command:

# Add capability for Microsoft Edge
$capabilitiesJson = "C:toolsseleniumedgeonlycapabilities.json"
@'
[
{
"browserName": "MicrosoftEdge",
"platform": "WINDOWS"
}
]
'@ | New-Item $capabilitiesJson -Type file -Force
choco install -y openjdk
choco install -y selenium --params "'/role:node /hub:localhost:4444 /javaoptions:-Dwebdriver.edge.driver=C:WindowsSystem32MicrosoftWebDriver.exe /capabilitiesJson:$capabilitiesJson /autostart'"

Finally, run Selenium WebDriver tests with the following capability:

// wdio.conf.ts Config file for WebdriverIO
const config: WebdriverIO.Config = {
capabilities: [
{
browserName: "MicrosoftEdge",
platformName: "WINDOWS",
"ms:inPrivate": true
};
]
}

"ms: inPrivate" capability allows you to open the InPrivate window and run tests.

When using the InPrivate window, browsing data (cookies, history, temporary files) will be deleted when the browser is closed.

It is recommended that this capability is enabled in order to eliminate unexpected dependencies between users, such as login status.

Challenges in the test environment

By following the above steps, we can run Selenium WebDriver tests on Windows 10 / Microsoft Edge.

At first we tried running these tests on a Windows PC that we had, but we soon faced several challenges.

The first challenge was execution time.

Using a single Windows PC took around 2 hours to run 120 test cases.

We could tolerate slow executing tests to a certain degree, as we did not need to run tests on Edge as frequently as other browsers like Chrome.

But ideally we wanted to keep the total execution time within 30 minutes.

We also felt that as we increased the number of parallel executions, maintaining more than one physical machine would become an issue in the future.

The next step then was to create the test environment in the cloud using Azure DevTest Labs.

Create a test environment on Azure DevTest Labs

Azure DevTest Labs is a service that enables provisioning and management of virtual machines as a development / testing environment.

DevTest Labs | Microsoft Azure

You can create a project (laboratory) and launch a VM by selecting the image (currently Windows 10 Pro, Version 1903) in MarketPlace.

You can reduce the cost of provisioning by using custom images and formulas when creating VMs with the same environment (this time Selenium Grid Node)

Create an Azure DevTest Labs custom image from a VM | Microsoft Docs

Manage formulas in Azure DevTest Labs to create VMs | Microsoft Docs

Azure DevTest Labs also has the feature to share the same public IP address with the VMs in the lab.

Understand shared IP addresses in Azure DevTest Labs | Microsoft Docs

For development / test environments, access restrictions using IP addresses are often applied.

By using this feature, it is possible to minimize the number of public IP addresses necessary for accessing individual VMs and accessing from individual VMs.

Finally, we created an environment where we could conduct 5 parallel tests of the Microsoft Edge browser on Azure DevTest Labs.

As a result, we were able to shorten the execution time from approximately 2 hours to 30 minutes or less.

Cost comparison with other services

Azure DevTest Labs service itself is free to use.

In addition to that, Windows licenses come for free when you use Azure DevTest Labs for development / testing purposes.

Therefore, charges for using Azure DevTest are usually limited to Azure resources such as Virtual Machine.

We felt that below VM sizes were enough to run Selenium WebDriver tests with Microsoft Edge browser.

• Standard_B1 ms (1 CPU, 2 GB RAM, ¥ 2,600 / month)
• Standard_B2s (2 CPU, 4GB RAM, ¥ 5,200 / month)

And finally, Azure DevTest Labs has the feature to automatically shutdown VMs when they are no longer used, as well as automatic startup when resources are requested.

This will allow us to run the VMs only when wanted the test to run, thus reducing costs.

Manage lab policies in Azure DevTest Labs | Microsoft Docs

There are SaaS such as SauceLabs / BrowserStack, which provide Microsoft Edge test environments in the cloud.

Here are some of their benefits so you can choose accordingly.

Benefits of SauceLabs / BrowserStack

• No need to create or maintain the environment

• Detailed versions of OS/Browser are prepared for the environment

Benefits of creating an environment with Azure DevTest Labs

• Once created, the number of parallel runs can be easily added and the cost will not increase so much.

  • Basically, Sauce Labs / BrowserStack is charged based on the number of concurrent parallel runs

• Able to create an execution environment in Japan (East Japan / West Japan)

  • SauceLabs / BrowserStack is currently not hosted in Japan region

• Because it’s IaaS, you can customize network and so on

Conclusion

In this article, I described how we deal with UI test automation (Selenium WebDriver) with Microsoft Edge browser, and how we create its environment using Azure DevTest Labs.

Previously we thought that creating testing environments for Windows 10 / Microsoft Edge would always be much harder, but Azure DevTest Labs made it relatively easy to do so.

Testing on Windows 10 / Microsoft Edge is still harder to do compared to using Chrome / Firefox. But for now we workaround these issues by controlling the frequency and the number of test cases that are run in these browsers.

This time, we used Azure DevTest Labs solely for the purpose of browser testing, but it can actually be used for other testing automation purposes.

For example, it is also possible to create a test automation environment for Windows App using Appium and Azure DevTest Lab.

Appium Pro: Testing Windows Desktop Apps With Appium

I will continue the challenge of building a test environment that is more stable and faster.