Open Source Vs. Commercial Automated Solutions For Accessibility Testing

Introduction

There are a plethora of mobile app accessibility automation tools available. But not every tool is infallible when testing for mobile accessibility. This raises several questions:

• Are Open Source solutions superior to commercial solutions?
• Should you prefer commercial solutions?
• Can you use both?

Find the answers to these questions here.

This blog introduces a contrasting perspective to allow the use of different solutions while being realistic. Information about the investigation's method, results, and the conclusion is also collected.

To further shed light on the topic, issues discovered by the Open Source Library AllyUITests compared to the commercial solution Evinced SDK for iOS are used. For Android, the focus will be on the issues uncovered by Native Library AccessibilityTestFramework (ATF) compared to the commercial solution Evinced SDK.

Getting Started With The Method

Leveraging the power of automated tools in testing is rewarding in many ways. But the community is still looking for a silver bullet to act as a complete solution provider.

The SDK is standard for iOS and Android, which is why it was compared with XCUITest, the AllyUITests Library for iOS, Espresso Test, and the ATF Library for Android. The application under test is the Sauce Labs's Catalog page.

Summary Of The Results

The results acquired for both Android and iOS are given below.

Android

The Evinced SDK discovered a total of 89 issues, 40 of which were violations, and 49 were possible issues that required human inspection. On the other hand, ATF found a total of 99 problems. Out of all the issues, 50 were violations, and 49 were possible problems that needed a manual inspection.

A total of 188 problems were discovered. And ATF found ten unique problems that Evinced SDK couldn’t.

iOS

For iOS, the Evinced SDK discovered a total of 74 issues, 40 of which were violations, and 34 were possible issues that required human inspection. The AllyUITests found 418 issues, 45 violations, and 374 warnings.

Both tools discovered 420 problems. AllyUITests found 344 warnings that the Evinced SDK missed. Both Evinced SDK and AllyUITests discovered two issues. As a result, a combined total of 492 issues were identified.

AllyUITests found more problems than Evinced did, but those detected issues were not relevant. An extensive analysis of these issues is given in the anomalies section.

Analyzing The Results

It's easy to observe that the tools report issues differently. For example, AllyUITests and ATF can only find problems, warnings, or nothing. Evinced, on the other hand, can find both actual and possible problems.

This evaluation assumes that Evinced has resolved the problem, even if it requests human verification. Further interpretation is required to understand:

• How the problems are outlined
• Ease of readability of the report
• Presence of screenshots
• The number of critical issues found

Interpreting Evinced SDK Results

The Evinced SDK can produce two types of reports: a JSON report and an advanced HTML report. Other reporting tools can leverage the JSON report to create meaningful reports in the environments. The JSON report contains a large amount of data to analyze.

The HTML report is comprehensive and includes several accessibility issues in a tabular form. The table has all the information required to troubleshoot and figure out the problem. The report also consists of the following features:

Accessibility Problem Category

Accessibility violations can be split into different categories depending on the issue. It includes all current accessibility rules like a tappable area, duplicate name, and accessible name. Clicking on the issue description will also give you a more descriptive understanding.

Severity

The severity category includes various problem types and their severity. It contains all current severity levels:

• Critical
• Serious
• Moderate
• Minor
• Best Practice
• Needs Review

The needs review category necessitates human intervention to validate the accuracy of the results.

Element

This is a single element for which the Evinced SDK identified at least one accessibility violation. Element type, accessibility label, and accessibility ID data points are also attached to this item.

Element Description

This section briefly summarizes the issue in a single sentence. Visual aids can also be used to place elements by looking at the screenshot attached, which shows various problems.

Interpreting Espresso + ATF Results

The report was compiled by using the Espresso and ATF accessibility libraries. After compilation, Gradle was used for formatting the report. It generates a basic report. Non-technical people will need help understanding the report because it only produces technical exceptions primarily for debugging.

Issues like image view ID, the element's height and width, and whether or not it is clickable, focusable, or selected are all included in the report. It also provides element coordinates and recommendations for fixing accessibility issues. Unfortunately, no screenshots or highlighted text are included in this report.

Interpreting XCUI Results

The report was compiled using the XCUI Test Framework and the AllyUITest accessibility library. After compilation, it was formatted using XCPretty to create a basic HTML report with the following information:

• Accessibility problem
• The criteria used to identify the problem as an accessibility issue
• Classification of the problem as either an error or a caution
  

This report does not include screenshots with the highlighted screen element. Also, no suggestions for fixing the issues are offered in this report.

Categorizing Problems

When tests are run for the Sauce Labs app’s catalogue page, the issues are grouped into different categories for different tools.

Classification Of Issues In iOS

The problems discovered by the AllyUITest Library and the Evinced Flow SDK can be categorized for easy comparison.

Classification Of Issues In Android

The issues found in Android by the ATF Library and the Evinced Flow SDK can be categorized for easier comparison.

Comparison Between The Solutions Based on Severity

Based on the report generated by each automated solution, accessibility issues can be categorized according to the severity of the issues.

For Android:

For iOS:

Anomalies

There are a few anomalies in the Evinced SDK and ATF framework. But the features that distinguish AllyUITest from other frameworks are:

Tappable Area

AllyUITests divides this type of problem into two subcategories for a given UI component, including:

• The interactive element is not tall enough
• The interactive element is not wide enough

Compared to other tools, this results in twice as many counts. This is because other tools generally state that the tappable area of the element is too small and does not further divide the category into two parts.

Duplicate Names

The experts at Axelerant think that the unusually high number of bugs found in this category could be due to a recently opened issue in which access to hidden elements is being tested.

Conclusion

It is difficult to prioritize one solution over the other as Open Source and commercial tools find similar problems with a few unique issues. Even if AllyUITest, or ATF, finds more bugs than the Evinced SDK, that doesn't mean it's necessarily superior, as the bugs covered by this suite of tests may play to the advantages of Open Source solutions. The Evinced SDK will likely beat Open Source Solutions on different tests. Also, the Evinced SDK has better reporting than native solutions, with easy-to-understand bugs, categories, and screenshots.

Depending on the user's needs and overall strategy, one should pick the preferred tools based on these tests. A hybrid approach involving fully automated solutions and supplementary semi-automated scanners is possible.

For example, if you use the Evinced SDK as part of a fully automated solution, you can use Accessibility Scanner and Accessibility Inspector as semi-automated tools to improve coverage. Alternatively, Open Source solutions like AllyUITest and the ATF libraries can be used for an automated solution and supplement that with the semi-automated Evinced Mobile Analyzer and Accessibility Scanner.

The second option has zero associated costs because it uses entirely free resources. But Evinced SDK requires a payment to be made.

Read more about Accessibility testing here