In this paper, we advocate for applying item response theory (a framework for evaluating exams) to a class of bias tests for language models.

One common way in which language models are tested for biases is with so-called bias benchmark datasets: collections of questions that are meant to reveal whether the model has a preference for stereotypical versus non-stereotypical language. For example, to test whether the model picked up on gender stereotypes, we might ask it several questions like whether “The doctor went home” is more similar to “The man went home” than to “The woman went home”.

Currently, AI researchers for the most part just collect as many of these test questions as they can and blindly throw them together. This is very different from how important exams for people (like the PISA exam) are constructed: For such tests, exam creators evaluate the quality of test items (e.g., check whether they can distinguish between people that are good vs bad at math) and make sure that the test items are not redundant (e.g., we don’t want 40 of our 50 math questions to be simple multiplication, else the exam will be too simple for many students).

Because the psychometric framework of item response theory (IRT) provides mathematical tools for evaluating how good and redundant test items are, it is often used for the creation and quality control of these important human exams. In this paper, we introduce the basics of IRT and argue that language model bias benchmark datasets should similarly be evaluated by and improved with IRT methods.

This paper is a product of our larger project on improving bias measurement tools.