Accessibility audit vs. accessibility testing

There are currently three different options available to you when testing a website for accessibility:

• Accessibility audit - An accessibility expert reviews your site, highlighting any accessibility issue
• Accessibility testing - Real disabled users complete common tasks on your website whilst a moderator notes all problems they experience
• Automated accessibility testing - An automated program evaluates your website against accessibility guidelines

Accessibility audit

An accessibility audit involves an accessibility expert reviewing your site, highlighting all accessibility issues and providing recommendations for fixing them. The reviewer would typically use assistive software used by disabled web users (e.g. a screen reader) to effectively carry out the audit, along with the Web Accessibility Toolbar.

You could hire an external accessibility consultancy to do this, but it's also possible to conduct the audit in-house. By reading through the W3C accessibility guidelines and attending a web accessibility training course (the latter to learn 'real-world' accessibility) you can gain a base level of accessibility knowledge. You should then be able to get your website up to a reasonable level of accessibility.

The main benefit of using an accessibility audit to evaluate your website is that accessibility audits are significantly cheaper and quicker than accessibility testing. Accessibility audits are often more comprehensive than accessibility testing in their depth and breadth of recommendations.

The main disadvantage of accessibility audits is that they're not designed for knowledge transfer. As such, your web team won't gain a great understanding of web accessibility nor are they likely to get much extra buy-in into accessibility. Both of these can be remedied through effective accessibility training.

Accessibility testing

There are a number of organisations in the accessibility world that swear by accessibility testing. The Disability Rights Commission, for example, have consistently said that testing a website with real disabled users is the only way to ensure it offers optimum accessibility. Their formal investigation into the accessibility of 1000 websites and the PAS 78 document they spearheaded both very strongly state that accessibility testing is the way to uncover all accessibility issues.

The main benefits of conducting accessibility testing include:

• You'll uncover website usability issues too
  As accessibility testing involves testing your website with real people, you'll get the side benefit of uncovering website usability issues too. If however you're going to invest the time and money involved in conducing accessibility testing then you might as well just carry out regular usability testing¹¹. Regular usability testing will uncover more usability issues as disabled users tend to take longer to complete tasks so carry out fewer tasks in a testing session.
  
• It's a great learning experience for your web team
  Running accessibility testing sessions, particularly if you're able to watch them live, is a great learning experience for everyone involved. There's nothing more eye opening than watching real users with real disabilities attempt to use your site. Accessibility testing is also a great way of getting the web team to get buy-in into accessibility.

There are also some disadvantages of using accessibility testing as a way of evaluating your website's accessibility:

• It's extremely expensive to do properly
  In order to carry out statistically valid accessibility testing you'll need to test with at least 5-6 users per user group. This means testing with 5-6 screen reader users, 5-6 screen magnifier users, 5-6 motor impaired users, 5-6 users with learning difficulties etc. Within these groups there are sub-groups (for example, there are different types of motor impairments) so to do proper accessibility testing you'll have to test with around 30 users.
  
• It'll find fewer problems than an accessibility audit
  Surprisingly, accessibility testing actually brings up fewer problems (and therefore fewer recommendations) than an accessibility audit. This is basically because real users often don't realise when a problem or nuisance occurs so don't voice their annoyance.
  
  For example, vertical bars are often used to separate horizontal navigation links, with each vertical bar being announced to screen reader users as “vertical bar”. Screen reader users are unlikely to comment or complain about this as they probably won't know what a 'vertical bar' is. An accessibility audit would however point this out as a problem and offer the solution (the solution would be to remove the vertical bars and insert them as right borders (through the CSS) on to the links).

Which accessibility evaluation technique should you use?

Generally speaking, it's not necessary to conduct accessibility testing most of the time. To conduct accessibility testing properly is extremely expensive and simply not worth the return on investment for most organisations. It's unlikely to highlight any major accessibility issues that don't come out of the accessibility audit (provided that the audit is carried out by an accessibility expert).

Are we really recommending to not do accessibility testing then? Well mostly, yes. Accessibility is far more guideline-driven than usability so accessibility solutions can easily be transferred from one website to the next. Provided that some organisations continue to carry out accessibility testing, you can 'piggy-back' off the knowledge that they've gained. By employing an accessibility expert to carry out an accessibility audit you'll quite quickly have this 'real-world' accessibility knowledge applied to your website.

If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented. There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit.

In reality though, you often only need to conduct accessibility testing if you want your web team to learn about accessibility and gain buy-in into it. Accessibility testing can also be useful if you wish to test complex interactions (e.g. Flash, complex JavaScript) that can sometimes fall beyond the scope of an accessibility audit.

But what about automated accessibility testing tools?

Generally speaking, automated accessibility testing tools are very limited in being able to uncover real problems. They can't adequately check for the majority of accessibility guidelines and can sometimes report problems that aren't actually problems. For more on the disadvantages of automated testing tools, read our article, The problem with automated testing tools.

The only real benefit of using an automated testing tool is to get a top level feel of how accessible (or inaccessible) your website is. In terms of getting real recommendations for fixing real problems then an accessibility audit is vastly superior.

Trenton Moss, Webcredible