Hello everyone! @vishal from Mercari’s SET team. This is the 12th entry for Mercari Advent Calendar. In this blog post, I will talk about rspec_html_reporter which is being used for Android E2E test automation at Mercari.

Role of Test Reports in Mobile E2E Test Automation

Test report is the end result of test automation execution. This is the only source of truth once test execution is finished. If a test is failed, developers only have test reports to decide whether it is because of a bug or test itself is flaky. So, A good test report should be a snapshot of whole test execution. It should contain various metadata of test execution to make debugging easier for failed test cases.

Unlike Unit Testing, where simple XML test report is enough as most of the time test only fails because of some issue in program logic. In case of Android E2E test, test can fail because of many reasons, such as:

• User Interface issues:
  • UI is not working as per feature specification.
  • Test code is not handling UI automation logic properly for scenarios involving scroll, swipe etc and test is flaky.
• Backend issues:
  • Backend is temporary down.
  • API is not working as per specification.
• 3rd party service issues:
  • Service is temporary down and error handling is not being done properly in test automation logic.
• Hardware issues:
  • Test is failing on specific device because of hardware limitation such as Memory, Processor.
  • Device wifi is unstable.
• Network issues:
  • Temporary network failure.

In the above list, it can be easily observed that there can be many reasons for a test to be failed. Some of them are persistent such as UI issue and some of them are non-persistent such as temporary network failure. Sometime test is failing because of bug and sometime because of test code issue. Without a proper report it is very difficult to know the cause of failure. And this is one of the main reason, why developers don’t trust mobile e2e tests much. And test always ends up unmaintained.

With a better test report, it is possible to gain this trust back. And rspec_html_reporter is written to try this idea.

Characteristics of an Ideal test reporter for Android E2E Test Automation

An ideal test report should contain all the metadata to debug failed tests easily. Following is a list of information which can be very useful for general android test scenario:

• Screenshot when test assertion is failed.
• Video of full test execution.
• ADB logcat for full test execution.
• Timestamp for each UI action and assertion. This can help in case problem is with backend and need to check server logs.

With a report containing above information, it will be very easy to know what is the reason of test failure.

rspec_html_reporter

rspec_html_reporter is extension of RSpec’s HTML formatter. It publishes pretty HTML report and add feature to embed metadata such as screenshot and screenrecord in HTML report. This is sample report which embed screenshots and screenrecord taken during test execution. Although, this reporter can be used for any kind of uitest such as Browser, Android, iOS etc. In this blog post I will mostly focus on Android side. Here is a sample implementation which uses Appium.

Rspec HTML Reports – Overview

github.com

Overview Page of Sample Report

How it works

Currently, rspec_html_reporter looks for following metadata after test is finished.

• @metadata[:screenshots]: Path of screenshots taken during test execution
• @metadata[:screenrecord]: Path of screenrecord taken during test execution
• @metadata[:failed_screenshot]: Path of screenshot taken when test assertion fails

Reporter will embed these files into HTML report if they are set during execution. It is test code job to create these metadata and set them in RSpec’s example (test case) :metadata param.

Note: I have plan to support adb logcat and UI action timestamp in this reporter soon.

Report Page for Failed Test Case

Adding screenshot and screenrecord in Android E2E tests.

RSpec provides hooks such as config.before and config.after which runs before and after each test cases. This is the best place to add metadata in test cases. As I have mentioned above, it is job of test code to create metadata. So, test code needs to capture screenshot and screenrecord. In case of android, we can easily do it using ADB. Please refer ScreenUtil module to understand how to take screenshots and screenrecord. Below are sample code snippet to add screenshot and screenrecord. They need to be written in spec_helper.rb file.

Adding screenshot when test is failed

  config.after do |example|
if example.exception
example.metadata[:failed_screenshot] = screenshot_rel_path(take_screenshot)
end
end

Above code will run after each test case and check if test is passed or failed. In case it is failed, it will take screenshot and add its path into :failed_screenshot metadata.

Adding screenrecord for each test

config.before do |example|
start_screenrecord
end
config.after do |example|
stop_screenrecord
example.metadata[:screenrecord] = screenrecord_rel_path(pull_screenrecord)
end

config.before block will run before each test case and start adb screenrecord server. config.after will run once test is finished. This will stop screenrecord and pull video from device storage and add its path to :screenrecord metadata.

Conclusion

Writing and troubleshooting mobile test automation is too time consuming because of longer execution time. This is one of the biggest reason why developers hate e2e tests. Having good report can improve developer productivity. Although, this reporter is written in Ruby and only works for RSpec, but important point here is understanding the core idea, i.e How Important Test Reports Are. I hope this will help you.

Tonight, we are having SRE-SET automation night at Mercari. I will talk about test automation reporting in more detail. There will be lots of automation related talks. Please join us if automation interests you.

connpass.com