About the AER Study

Kessler, Low, and Sullivan – An example IRR Study

Understanding employer preferences and discrimination in labor markets has been of fundamental interest to economists, social scientists, and policy makers. To shed new light on these important research questions, we pioneered Incentivized Resume Rating (IRR), in Kessler, Low, and Sullivan (Nov 2019, American Economic Review, lead article).

In an IRR study, employers are asked to rate hypothetical resumes. Based on their responses, employers are matched to real job applicants. Because the algorithm helps employers find promising applicants, employers are incentivized to seriously consider the hypothetical resumes and report their true preferences. The employer incentives distinguish IRR from a typical unincentivized survey.

Kessler, Low, and Sullivan first implemented IRR with a group of employers looking to hire University of Pennsylvania graduates. IRR revealed evidence that STEM employers discriminate against women and minority candidates: on average women and minority candidates needed an additional 0.22 GPA points to be rated equivalently to otherwise identical male resumes.

IRR’s ability to elicit a rich set of response variables also revealed a previously undocumented channel of discrimination. Specifically, employers were asked to rate both how qualified a given applicant was for a position and how likely the applicant was to accept the position. Women were rated as less likely to accept job offers. This finding suggests that employers may choose not to offer jobs to women they otherwise consider qualified out of a concern that those women will not accept the job. This potential channel for discrimination had not previously been documented in the literature.

IRR was conceived as an alternative to traditional resume audit studies. In traditional resume audit studies, researchers send fictitious resumes to employers and infer their preferences by measuring which resumes receive interview call-backs. IRR has a number of advantages relative to audit studies. First, IRR allows researchers to openly cooperate with employers by providing a service that employers value — rather than relying on deception and inducing employer participation without their consent. Second, IRR allows employers to rate resumes on multiple dimensions, using more continuous measures, whereas audit studies are usually restricted to measuring employer call-back rates. Third, resume audit studies can only be used to study those labor markets where employers consider resumes sent to them from the general public. However, IRR can be used with any group of employers interested in benefiting from being matched to job seekers.

We believe IRR can help researchers more broadly investigate a wide range of questions about discrimination and labor economics. We have created this website to share the IRR tool with researchers considering using IRR in their projects. (To see those materials and get started, click here.)