Evaluating Automated Accessibility Checker Tools

The internet has grown into a tool that is essential for communication, education, and commerce. However, for those with disabilities, web content can often have significant barriers if it is not designed with accessibility in mind. Digital accessibility ensures that websites and applications are usable by everyone, regardless of ability. In order to make sure web content complies with accessibility standards, automated accessibility evaluation tools have grown in significance. This paper investigates the effectiveness of automated accessibility evaluation tools (AAETs) for identifying web accessibility issues. The research analyzed five prominent AAETs: Accessibility Insights, aXe DevTools, Lighthouse, SiteImprove, and WAVE and tested them against four popular news websites and one local news site. The paper introduced a weighted coverage error ratio (CER) metric which benefits from all issues found regardless of accuracy. After analyzing the results, WAVE was the most effective tool in the scope of this paper if only by a small margin. Ultimately, the best way to evaluate accessibility is to use a combination of the tools and incorporate manual testing.