Accessibility audit vs. accessibility testing

This is a very interesting article and I agree with much of what Trenton Moss has said. I would however, like to take issue with one or two points he makes:

Trenton says “The Disability Rights Commission… have consistently said that testing a website with real disabled users is the only [my italics] way to ensure it offers optimum accessibility… the PAS 78 document… very strongly state that accessibility testing is the way to uncover all [my italics] accessibility issues..”

I am not an apologist for the DRC but that is not my reading of their position. PAS 78 says “Comprehensive evaluation of web site accessibility should involve a combination [my italics] of conformance with the technical requirements of WCAG, and user testing of accessibility features.” PAS 78 talks extensively about expert reviews and conformance inspection – why these are useful and how they should be done. I would also be surprised if they, or anyone else, would argue than any one approach or even any combination of approaches would identify all issues.

As far as I can see they, and I, would not disagree with Trenton’s statement “If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented” – this is eminently sensible, as he says “There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit”

However, I, (and I am sure the DRC) would not agree with his statement “[user testing is] unlikely to highlight any major accessibility issues that don't come out of the accessibility audit.” Having conducted lots of disabled user tests after thorough accessibility audits (done by very competent accessibility consultants) I can assure Trenton that “Oh yes it will!” For example in a recent disabled user testing session for a well know supermarket client we identified over 10 serious issues (issues that would prevent a user achieving their goals) not identified in the audit. One of these included not being able to find the ‘Add to basket’ button!

A key reason why expert assessments fail to find all serious issues is the same whether it’s a usability or accessibility audit. The expert can never be expert enough! It is almost impossible for a reviewer to be fully aware of how the user thinks and will behave. How can a male, 30 something, HCI expert get inside the head of a blind, wheel chair bound woman of 40 with a motor neuron disease? Having watched many hundreds of people on web sites I am now less likely to predict how I think users will behave than earlier in my career. My view of the world is inevitably influenced by the fact I am male, white and 50+! The other benefit of testing with disabled users is that it identifies usability issues not identified by testing with able bodied users – but do effect able bodies users. It is my view, and clearly that of the DRC, that only by testing with disabled users, as well as undertaking an accessibility audit can you be sure you have identified the serious accessibility (and usability) issues.

Peter Collins - Web Usability Partnership