Implementing WCAG 2.0 Guidelines through Automated Testing

Any successful web site is in a continual state of flux. New content, new functions, changes to the structure of the site and even changes to the supporting infrastructure occur on a daily, if not second-by-second, basis.

For the users and the marketeers this continual change is seen as a major benefit of web technology. But for anyone concerned about ensuring that a web site is compliant, and in particular accessible, the changes to the site and its environment create a challenge. Audits, automated tests, manual inspections and user testing are all excellent methods of checking for compliance. They can ensure that the site is compliant at the time of the check. However, any subsequent change may make the site non-compliant; after one month the probability is that the site will not be compliant, any longer and the number of changes will ensure that the site is non-compliant.

The continual flux of websites means that there is a need for continual compliance checking.

This has been brought home to me when I ran the latest version (3.2) of HiSoftware Compliance Sheriff against a website I helped to develop. The reports produced showed that the site had accessibility, privacy and structural errors in it. These had slipped in because:

• Small changes to the infrastructure had been made but had not been checked for compliance.
• New content had been added and some simple coding errors had slipped through.
• Links to other web sites remained even though the target pages no longer existed.
• The recommendation for Privacy best practice had been upgraded but the site had not been updated to reflect these changes.
• A new version of the Web Content Accessibility Guidelines had been published (WCAG 2.0) and again these improvements had not been reflected in the web site.

I am sure that this is a common story for many web sites.

The good news is that Compliance Sheriff is user friendly and the reports are easy to understand. The flexibility allows for dashboard reporting to be used by management to quickly analyse site results at a high level. Actionable reports are created through a drill-down link feature, where it is simple for a developer to see exactly what each problem is, where it is and how to fix it through the highlighting of the source code. The product also allows users to view the rendered page directly from the report with areas of non-compliance outlined by a red box for simple identification.

Even though it is an easy product to use, Compliance Sheriff is full-featured and there are a variety of options to choose from in the initial stages of set up. Hence HiSoftware recommends that a consultant works with the users to develop the initial set up and customisation. This service is provided directly by HiSoftware in North America and by partners elsewhere including Europe and Apac; HiSoftware has just signed an agreement with AbilityNet to provide these services in the UK.

During the initial set up, users can decide exactly what checks will be carried out and where. It can also remove individual checks (for example the implementation of Google Analytics on my site triggered errors on every page, however the issue is not serious and need not be tackled immediately so the scans need to be configured to ignore this issue). All the checks are defined using a simple scripting language (regular expressions) so new ones can be added to reflect specific rules for a web site—for example, these could include rules about the size and positioning of logos.

The major enhancement in Compliance Sheriff 3.2 is support for the new WCAG 2.0 standards. The relationship between the tests performed and guidelines is very easy to follow.

WCAG 2.0 is based on the four principles that the web pages should be perceivable, operable, understandable and robust (POUR for short). These principles are divided into 12 guidelines and are supported by success criterion; each guideline has a description, an explanation of its importance, a set of techniques for implementation, and a set of tests to validate compliance. This hierarchy is directly reflected in the implementation of the tests performed.

To take one example:

• Principle 1 “Perceivable”.
• Guideline 1.3 “Adaptable: Create content that can be presented in different ways without losing information or structure”.
• Criteria 1.3.1 “Info and Relationships: Information, structure, and relationships conveyed through presentation can be programmatically determined or are available in text”.
• Technique H39 “Using caption elements to associate data table captions with data tables”.
• Test “check that a data table has a caption”.

Which in brief says for a data table to be perceivable by a user of a screen reader then it must include a caption.

In Compliance Sheriff there is a group “WCAG 2.0 – Criterion 1.3.1 [Info and Relationships]” which includes a checkpoint “Accessibility20 H39: Use caption elements to associate data table captions with data tables” and its script says “check that each data table has a non-blank caption”.

This transparent reflection of WCAG 2.0 in the Compliance Sheriff tests makes clear both the strengths of the solution and limitations of an automated tool.

Its strengths are that it can automatically check any rules that can be defined in a script. It does this quickly, efficiently and accurately across a large website and across related sites. This is very difficult to do manually and will, as I discovered in my tests, expose many issues with existing sites.

As with any automated solutions, the limitations are that the guidelines include tests that cannot be defined in a computer script. For example, the test for Technique H39 also says ‘the wording of the caption should identify the table’. Only a human can validate that the wording is suitable so it cannot be automated.

This means that manual checks must be included in the continuing verification of any website. However, I believe that if a site passes the automated tests it is a strong indication that the site has been well constructed and therefore it will also pass the manual verification. If, on the other hand, the site fails the automated tests then it is certain that there are problems and during the remediation process further manual tests need to be carried out.

Although Compliance Sheriff cannot automate the manual checks, it does provide assistance by providing a list of visual checks that need to be completed and the facility to report the results of the tests. For each automated and manual test the results wizard shows if they have passed, failed, are not applicable, have a warning or if they require a visual check. The tester can then revise the results, possibly setting a failure to N/A or by changing a visual check to passed, each of these changes to the results is marked with the id of the person making the change. In this way all test results are consolidated and presented on one report; including failures and warnings that require further work by development, as well as visual checks that have been successful.

My suggested procedure for manual and automated testing is:

• During early prototype and design: use the automated tool to remove simple errors.
• In later prototype and design: run user tests to ensure compliance and usability.
• During development: use automated testing and good process control to keep a high level of compliance and usability.
• Before moving to production: run further user-centric tests.
• Post-production: schedule regular automated tests, if the results are good then manual testing may not be needed.
• Run intermittent user testing when automated testing shows a degradation of the site or when user feedback suggests there is a problem.

This mixture will provide a good balance between levels of compliance and cost and effort.

Compliance Sheriff provides a highly effective and transparent testing solution, which fits neatly into the overall development and testing regime required to make an organisation’s web presence more accessible and compliant.