On May 12, 2022, the Equal Employment Opportunity Commission (EEOC) issued technical guidance on how employers’ use of artificial intelligence (AI) and other software tools to make employment decisions may violate the Americans with Disabilities Act (ADA). The U.S. Department of Justice (DOJ) also issued similar guidance the same day.
The ADA, which applies to employers with 15 or more employees, prohibits discrimination based on disability and requires reasonable accommodations to allow qualified individuals with disabilities to be evaluated for or perform a job. The EEOC’s guidance on AI explains how, in the absence of safeguards, an employer’s use of certain software tools to select new employees, monitor performance, determine pay or promotions, or administer or score tests may violate these ADA provisions.
This Compliance Overview provides the EEOC’s guidance for employers.
LINKS AND RESOURCES
- Text of the EEOC’s guidance, “The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees”
- Text of the DOJ’s guidance, “Algorithms, Artificial Intelligence, and Disability Discrimination in Hiring”
HIGHLIGHTS
SOFTWARE USE AND THE ADA
- »Employers using software tools to make employment decisions must ensure the tools do not unfairly rate or screen out qualified individuals with disabilities.
- »Reasonable accommodations may be required to ensure an otherwise-qualified individual’s disability does not cause software tools to screen out or unfairly rate the individual.
- »Employers must also ensure their use of software tools does not result in unlawful disability-related inquiries or medical examinations.
PROMISING PRACTICES
- The EEOC guidance outlines several recommendations for reducing the risk of software-related ADA violations.
ARTIFICIAL INTELLIGENCE
Employers now have a wide variety of computer-based tools available to assist them in hiring workers, monitoring worker performance, determining pay or promotions, and establishing the terms and conditions of employment. Employers may utilize these tools to save time and effort, increase objectivity or decrease bias. However, the use of these tools may disadvantage job applicants and employees with disabilities. When this occurs, employers may risk violating federal equal employment opportunity (EEO) laws that protect individuals with disabilities.
The questions and answers in this document explain how employers’ use of software that relies on algorithmic decision-making may violate existing requirements under the ADA. This technical assistance also provides practical tips to employers on how to comply with the ADA.
The EEOC enforces and provides leadership and guidance on the federal EEO laws prohibiting employment discrimination on the basis of race, color, national origin, religion and sex (including pregnancy, sexual orientation, and gender identity), disability, age (over 40) and genetic information. This publication is part of an ongoing effort by the EEOC to educate employers, employees and other stakeholders about the application of EEO laws when employers use employment software and applications, some of which incorporate algorithmic decision-making.
BACKGROUND: SOFTWARE, ALGORITHMS AND ARTIFICIAL INTELLIGENCE
As a starting point, this section explains the meaning of three central terms used in this document—software, algorithms and artificial intelligence (AI)—and how they relate to each other when used in a workplace.
Software
Broadly,“software” refers to information technology programs or procedures that provide instructions to a computer on how to perform a given task or function. “Application software” (also known as an “application” or “app”) is a type of software designed to perform or to help the user perform a specific task or tasks.
There are many different types of software and applications used in employment. These include automatic resume-screening software, hiring software, chatbot software for hiring and workflow, video interviewing software, analytics software, employee monitoring software and worker management software.
Algorithms
Generally, an “algorithm” is a set of instructions that a computer can follow to accomplish some end. Human resources software and applications use algorithms to allow employers to process data to evaluate, rate and make other decisions about job applicants and employees. Software or applications that include algorithmic decision-making tools may be used at various stages of employment, including hiring, performance evaluation, promotion and termination.
AI
Some employers and software vendors use AI when developing algorithms that help employers evaluate, rate and make other decisions about job applicants and employees. In the National Artificial Intelligence Initiative Act of 2020, Congress defined AI to mean a “machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments.”
In the employment context, using AI has typically meant that the developer relies partly on the computer’s own analysis of data to determine which criteria to use when making employment decisions. AI may include machine learning, computer vision, natural language processing and understanding, intelligent decision support systems, and autonomous systems.
Employers may rely on different types of software that incorporate algorithmic decision-making at a number of stages of the employment process. Examples include:
- »Resume scanners that prioritize applications using certain keywords;
- »Employee monitoring software that rates employees on the basis of their keystrokes or other factors;
- »“Virtual assistants” or “chatbots” that ask job candidates about their qualifications and reject those who do not meet predefined requirements;
- »Testing software that provides “job fit” scores for applicants or employees regarding their personalities, aptitudes, cognitive skills or perceived “cultural fit” based on their performance on a game or on a more traditional test.
Each of these types of software may include AI.
ADA BASICS
1. What is the ADA, and how does it define “disability”?
The ADA is a federal civil rights law. Title I of the ADA prohibits employers, employment agencies, labor organizations and joint labor-management committees with 15 or more employees from discriminating on the basis of disability. Other parts of the ADA, not discussed here, ensure that people with disabilities have full access to public and private services and facilities.
The ADA has a very specific definition of a current “disability.” A physical or mental impairment meets the ADA’s definition of a current “disability” if it would “substantially limit” one or more “major life activities” when left untreated. Major life activities include, for example, seeing, reaching, communicating, speaking, concentrating or the operation of major bodily functions, such as brain or neurological functions.
A condition does not need to be permanent or severe or cause a high degree of functional limitation to be “substantially limiting.” It may qualify as substantially limiting, for example, by making activities more difficult, painful or time-consuming to perform compared to the way that most people perform them. In addition, if the symptoms of the condition come and go, the condition still will qualify as a disability if it substantially limits a major life activity when active. Many common and ordinary medical conditions will qualify.
2. How could an employer’s use of algorithmic decision-making tools violate the ADA?
The most common ways that an employer’s use of algorithmic decision-making tools could violate the ADA are:
- The employer does not provide a “reasonable accommodation” that is necessary for a job applicant or employee to be rated fairly and accurately by the algorithm.
- The employer relies on an algorithmic decision-making tool that intentionally or unintentionally “screens out” an individual with a disability, even though that individual is able to do the job with reasonable accommodation. “Screen out” occurs when a disability prevents a job applicant or employee from meeting—or lowers their performance on—a selection criterion, and the applicant or employee loses a job opportunity as a result. A disability could have this effect by, for example, reducing the accuracy of the assessment, creating special circumstances that have not been taken into account or preventing the individual from participating in the assessment altogether.
- The employer adopts an algorithmic decision-making tool for use with its job applicants or employees that violates the ADA’s restrictions on disability-related inquiries and medical examinations.
An employer’s use of an algorithmic decision-making tool may be unlawful for one of the above reasons or for several such reasons.
3. Is an employer responsible under the ADA for its use of algorithmic decision-making tools even if the tools are designed or administered by another entity, such as a software vendor?
In many cases, yes. For example, if an employer administers a preemployment test, it may be responsible for ADA discrimination if the test discriminates against individuals with disabilities, even if the test was developed by an outside vendor.
In addition, employers may be held responsible for the actions of their agents, which may include entities such as software vendors, if the employer has given them authority to act on the employer’s behalf
ALGORITHMIC DECISION-MAKING TOOLS AND REASONABLE ACCOMMODATION
4. What is a reasonable accommodation?
A reasonable accommodation is a change in the way things are done that helps a job applicant or employee with a disability apply for a job, do a job or enjoy equal benefits and privileges of employment. Examples of reasonable accommodations may include specialized equipment, alternative tests or testing formats, permission to work in a quiet setting, and exceptions to workplace policies. These are just examples—almost any change can be a reasonable accommodation—although an employer never has to lower production or performance standards or eliminate an essential job function as a reasonable accommodation.
5. May an employer announce generally (or use software that announces generally) that reasonable accommodations are available to job applicants and employees who are asked to use or be evaluated by an algorithmic decision-making tool and invite them to request reasonable accommodations when needed?
Yes. An employer may tell applicants or employees what steps an evaluation process includes and may ask them whether they will need reasonable accommodations to complete it. For example, if a hiring process includes a video interview, the employer or software vendor may tell applicants that the job application process will involve a video interview and provide a way to request a reasonable accommodation. Doing so is a “promising practice” to avoid violating the ADA.
6. When an employer uses algorithmic decision-making tools to assess job applicants or employees, does the ADA require the employer to provide reasonable accommodations?
If an applicant or employee tells the employer that a medical condition may make it difficult to take a test or may cause an assessment result that is less acceptable to the employer, the applicant or employee has requested a reasonable accommodation. To request an accommodation, it is not necessary to mention the ADA or use the phrase “reasonable accommodation.”
Under the ADA, employers need to respond promptly to requests for reasonable accommodation. If it is not obvious or already known whether the requesting applicant or employee has an ADA disability and needs a reasonable accommodation because of it, the employer may request supporting medical documentation. When the documentation shows that a disability might make a test more difficult to take or that it might reduce the accuracy of an assessment, the employer must provide an alternative testing format or a more accurate assessment of the applicant’s or employee’s skills as a reasonable accommodation, unless doing so would involve significant difficulty or expense (also called “undue hardship”).
For example, a job applicant who has limited manual dexterity because of a disability may report that they would have difficulty taking a knowledge test that requires the use of a keyboard, trackpad or another manual input device. Especially if the responses are timed, this kind of test will not accurately measure this particular applicant’s knowledge. In this situation, the employer would need to provide an accessible version of the test (for example, one in which the applicant is able to provide responses orally rather thanmanually) as a reasonable accommodation unless doing so would cause undue hardship. If it is not possible to make the test accessible, the ADA requires the employer to consider providing an alternative test of the applicant’s knowledge as a reasonable accommodation, barring undue hardship.
Other examples of reasonable accommodations that may be effective for some individuals with disabilities include extended time or an alternative version of the test, including one that is compatible with accessible technology (like a screen-reader) if the applicant or employee uses such technology. Employers must give individuals receiving reasonable accommodation equal consideration with other applicants or employees not receiving reasonable accommodations.
The ADA requires employers to keep all medical information obtained in connection with a request for reasonable accommodation confidential. It must store all such information separately from the applicant’s or employee’s personnel file.
7. Is an employer responsible for providing reasonable accommodations related to the use of algorithmic decision-making tools, even if the software or application is developed or administered by another entity?
In many cases, yes. As explained in Question 3 above, an employer may be held responsible for the actions of other entities, such as software vendors, the employer has authorized to act on its behalf. For example, if an employer were to contract with a software vendor to administer and score on its behalf a preemployment test, the employer likely would be held responsible for actions that the vendor performed—or did not perform—on its behalf. Thus, if an applicant were to tell the vendor that a medical condition was making it difficult to take the test (which qualifies as a request for reasonable accommodation), and the vendor did not provide an accommodation required under the ADA, the employer likely would be responsible even if it was unaware that the applicant reported a problem to the vendor.
ALGORITHMIC TOOLS THAT SCREEN OUT QUALIFIED INDIVIDUALS WITH DISABILITIES
8. When is an individual “screened out” because of a disability, and when is screening out potentially unlawful?
Screen out occurs when a disability prevents a job applicant or employee from meeting—or lowers their performance on—a selection criterion, and the applicant or employee loses a job opportunity as a result. The ADA says that a screening out is unlawful if the individual who is screened out is able to perform the essential functions of the job with a reasonable accommodation if one is legally required.
9. Could algorithmic decision-making tools screen out an individual because of a disability? What are some examples?
Yes, an algorithmic decision-making tool could screen out an individual because of a disability if the disability causes that individual to receive a lower score or an assessment result that is less acceptable to the employer, and the individual loses a job opportunity as a result.
An example of a screen-out might involve a chatbot, which is software designed to engage in communications online and through texts and emails. A chatbot might be programmed with a simple algorithm that rejects all applicants who, during the course of their “conversation” with the chatbot, indicate that they have significant gaps in their employment history. If a particular applicant had a gap in employment, and if the gap had been caused by a disability (for example, if the individual needed to stop working to undergo treatment), then the chatbot may function to screen out that person because of the disability.
Another kind of screen-out may occur if a person’s disability prevents the algorithmic decision-making tool from measuring what it is intended to measure. For example, video interviewing software that analyzes applicants’ speech patterns in order to reach conclusions about their ability to solve problems is not likely to score an applicant fairly if the applicant has a speech impediment that causes significant differences in speech patterns. If such an applicant is rejected because the applicant’s speech impediment resulted in a low or unacceptable rating, the applicant may effectively have been screened out because of the speech impediment.
10. Some algorithmic decision-making tools may say that they are “bias-free.” If a particular tool makes this claim, does that mean that the tool will not screen out individuals with disabilities?
When employers (or entities acting on their behalf, such as software vendors) say that they have designed an algorithmic decision-making tool to be “bias-free,” it typically means that they have taken steps to prevent a type of discrimination known as “adverse impact” or “disparate impact” discrimination under Title VII based on race, sex, national origin, color or religion. This type of Title VII discrimination involves an employment policy or practice that has a disproportionately negative effect on a group of individuals who share one of these characteristics, like a particular race or sex.
To reduce the chances that the use of an algorithmic decision-making tool results in disparate impact discrimination on bases like race and sex, employers and vendors sometimes use the tool to assess subjects in different demographic groups and then compare the average results for each group. If the average results for one demographic group are less favorable than those of another (for example, if the average results for individuals of a particular race are less favorable than the average results for individuals of a different race), the tool may be modified to reduce or eliminate the difference.
The steps taken to avoid that kind of Title VII discrimination are typically distinct from the steps needed to address the problem of disability bias. If an employer or vendor were to try to reduce disability bias in the way described above, doing so would not mean that the algorithmic decision-making tool could never screen out an individual with a disability. Each disability is unique. An individual may fare poorly on an assessment because of a disability and be screened out as a result, regardless of how well other individuals with disabilities fare on the assessment. Therefore, to avoid screen-out, employers may need to take different steps beyond the steps taken to address other forms of discrimination.
11. Screen-out because of a disability is unlawful if the individual who is screened out is able to perform the essential functions of the job with a reasonable accommodation if one is legally requir individual is screened out by an algorithmic decision-making tool, is it still possible the individual is able to perform the essential functions of the job?
In some cases, yes. For example, some employers rely on “gamified” tests, which use video games to measure abilities, personality traits and other qualities to assess applicants and employees. If a business requires a 90% score on a gamified assessment of memory, an applicant who is blind and therefore cannot play these particular games would not be able to score 90% on the assessment and would be rejected. But the applicant still might have a very good memory and be perfectly able to perform a job’s essential functions that require a good memory.
Even an algorithmic decision-making tool that has been “validated” for some purposes might screen out an individual who is able to perform well on the job. To say that a decision-making tool has been “validated” means that there is evidence meeting certain professional standards showing that the tool accurately measures or predicts a trait or characteristic that is important for a specific job. Algorithmic decision-making tools may be validated in this sense and still be inaccurate when applied to particular individuals with disabilities. For example, the gamified assessment of memory may be validated because it has been shown to be an accurate measure of memory for most people in the general population, yet still screen out particular individuals who have good memories but are blind and who therefore cannot see the computer screen to play the games.
An algorithmic decision-making tool may also sometimes screen out individuals with disabilities who could do the job because the tool does not take into account the possibility that such individuals are entitled to reasonable accommodations on the job. Algorithmic decision-making tools are often designed to predict whether applicants can do a job under typical working conditions. But people with disabilities do not always work under typical conditions if they are entitled to reasonable on-the-job accommodations.
For example, some preemployment personality tests are designed to look for candidates similar to the employer’s most successful employees—employees who most likely work under conditions that are typical for that employer. Someone who has post-traumatic stress disorder(PTSD) might be rated poorly by one of these tests if the test measures a trait that may be affected by that particular individual’s PTSD, such as the ability to ignore distractions. Even if the test is generally valid and accurately predicts this individual would have difficulty handling distractions under typical working conditions, it might not accurately predict whether the individual still would experience those same difficulties under modified working conditions—specifically, conditions in which the employer provides required reasonable on-the-job accommodations such as a quiet workstation or permission to use noise-canceling headphones. If such a person were to apply for the job and be screened out because of a low score on the distraction test, the screen-out might be unlawful under the ADA. Some individuals who may test poorly in certain areas due to a medical condition may not even need a reasonable accommodation to perform a job successfully.
12. What could an employer do to reduce the chances that algorithmic decision-making tools will screen out someone because of a disability, even though that individual is able to perform the essential functions of the job (with a reasonable accommodation if one is legally required)?
First, if an employer is deciding whether to rely on an algorithmic decision-making tool developed by a software vendor, it may want to ask the vendor whether the tool was developed with individuals with disabilities in mind. Some possible inquiries about the development of the tool that an employer might consider include but are not limited to:
- If the tool requires applicants or employees to engage a user interface, did the vendor make the interface accessible to as many individuals with disabilities as possible?
- Are the materials presented to job applicants or employees in alternative formats? If so, which formats? Are there any kinds of disabilities for which the vendor will not be able to provide accessible formats, in which case the employer may have to provide them (absent undue hardship)?
- Did the vendor attempt to determine whether the use of the algorithm disadvantages individuals with disabilities? For example, did the vendor determine whether any of the traits or characteristics measured by the tool are correlated with certain disabilities?
If an employer is developing its own algorithmic decision-making tool, it could reduce the chances of an unintentional screen-out by taking the same considerations into account during its development process. Depending on the type of tool in question, reliance on experts on various types of disabilities throughout the development process may be effective. For example, if an employer is developing preemployment tests that measure personality, cognitive or neurocognitive traits, it may be helpful to employ psychologists, including neurocognitive psychologists, throughout the development process in order to spot ways in which the test may screen out people with autism or cognitive, intellectual or mental health-related disabilities.
Second, regardless of whether the employer or another entity is developing an algorithmic decision-making tool, the employer may be able to take additional steps during implementation and deployment to reduce the chances that the tool will screen out someone because of a disability, either intentionally or unintentionally. Such steps include:
- Clearly indicating that reasonable accommodations, including alternative formats and alternative tests, are available to people with disabilities;
- Providing clear instructions for requesting reasonable accommodations; and
- Providing all job applicants and employees undergoing assessment by the algorithmic decision-making tool with as much information about the tool as possible in advance of the assessment. This may include providing information about which traits or characteristics the tool is designed to measure, the methods by which those traits or characteristics are to be measured and the disabilities, if any, that might potentially lower the assessment results or cause screen-out.
Taking these steps will provide individuals with disabilities an opportunity to decide whether a reasonable accommodation may be necessary. For example, suppose that an employer uses an algorithm to evaluate its employees’ productivity, and the algorithm takes into account the employee’s average number of keystrokes per minute. If the employer does not inform its employees it is using this algorithm, an employee who is blind or has a visual impairment and uses voice recognition software instead of a keyboard may be rated poorly and lose out on a promotion or other job opportunity as a result. However, if the employer informs its employees that they will be assessed partly based on keyboard usage, that same employee would know to request an alternative means of measuring productivity—perhaps one that takes into account the use of voice recognition software rather than keystrokes—as a reasonable accommodation.
Another way for employers to avoid ADA discrimination when using algorithmic decision-making tools is to try to ensure no one is screened out unless they are unable to do the job, even when provided with reasonable accommodations. A promising practice is to only develop and select tools that measure abilities or qualifications that are truly necessary for the job—even for people entitled to reasonable on-the-job accommodation. For example, an employer who is hiring cashiers might want to ensure that the chatbot software it is using does not reject applicants who are unable to stand for long periods. Otherwise, a chatbot might reject an applicant who uses a wheelchair and may be entitled to a lowered cash register as a reasonable accommodation.
As a further measure, employers may wish to avoid using algorithmic decision-making tools that do not directly measure necessary abilities and qualifications for performing a job but instead make inferences about those abilities and qualifications based on characteristics that are correlated with them. For example, if an open position requires the ability to write reports, the employer may wish to avoid algorithmic decision-making tools that rate this ability by measuring the similarity between an applicant’s personality and the typical personality of currently successful report writers. By doing so, the employer lessens the likelihood of rejecting someone who is good at writing reports but whose personality, because of a disability, is uncommon among successful report writers.
ALGORITHMIC DECISION-MAKING TOOLS AND DISABILITY-RELATED INQUIRIES AND MEDICAL EXAMINATIONS
13. How could an employer’s use of algorithmic decision-making tools violate ADA restrictions on disability-related inquiries and medical examinations?
An employer might violate the ADA if it uses an algorithmic decision-making tool that poses “disability-related inquiries” or seeks information that qualifies as a “medical examination” before giving the candidate a conditional offer of employment. This type of violation may occur even if the individual does not have a disability.
An assessment includes “disability-related inquiries” if it asks job applicants or employees questions that are likely to elicit information about a disability or directly asks whether an applicant or employee is an individual with a disability. It qualifies as a “medical examination” if it seeks information about an individual’s physical or mental impairments or health.
An algorithmic decision-making tool that could be used to identify an applicant’s medical conditions would violate these restrictions if it were administered prior to a conditional offer of employment. However, not all algorithmic decision-making tools that ask for health-related information are “disability-related inquiries or medical examinations.” For example, a personality test is not posing “disability-related inquiries” because it asks whether the individual is “described by friends as being ‘generally optimistic,’” even if being described by friends as generally optimistic might somehow be related to some kinds of mental health diagnoses.
Note, however, that even if a request for health-related information does not violate the ADA’s restrictions on disability-related inquiries and medical examinations, it still might violate other parts of the ADA. For example, if a personality test asks questions about optimism, and if someone with major depressive disorder (MDD) answers those questions negatively and loses an employment opportunity as a result, the test may “screen out” the applicant because of MDD. As explained in Questions 8-11 above, such screen-out may be unlawful if the individual screened out can perform the job’s essential functions, with or without reasonable accommodation.
Once employment has begun, disability-related inquiries may be made, and medical examinations may be required only if they are legally justified under the ADA.
PROMISING PRACTICES FOR EMPLOYERS
14. What can employers do to comply with the ADA when using algorithmic decision-making tools?
As discussed in Questions 4-7 above, employers must provide reasonable accommodations when legally required. Promising practices that may help employers to meet this requirement include:
- Training staff to recognize and process requests for a reasonable accommodation as quickly as possible, including requests to retake a test in an alternative format or to be assessed in an alternative way after the individual has already received poor results.
- Training staff to develop or obtain alternative means of rating job applicants and employees when the current evaluation process is inaccessible or otherwise unfairly disadvantages someone who has requested a reasonable accommodation because of a disability.
- Asking the entity to forward all requests for accommodation promptly to be processed by the employer in accordance with ADA requirements if the algorithmic decision-making tool is administered by an entity with authority to act on the employer’s behalf, such as a testing company. Alternatively, the employer could seek to enter into an agreement with the third party requiring it to provide reasonable accommodations on the employer’s behalf, in accordance with the employer’s obligations under the ADA.
Employers should minimize the chances that algorithmic decision-making tools will disadvantage individuals with disabilities, either intentionally or unintentionally. Promising practices include:
- Using algorithmic decision-making tools that have been designed to be accessible to individuals with as many different kinds of disabilities as possible, thereby minimizing the chances that individuals with different kinds of disabilities will be unfairly disadvantaged in the assessments. User testing is a promising practice.
- Informing all job applicants and employees who are being rated that reasonable accommodations are available for individuals with disabilities and providing clear and accessible instructions for requesting such accommodations.
- Describing, in plain language and in accessible formats, the traits that the algorithm is designed to assess, the method by which those traits are assessed, and the variables or factors that may affect the rating.
Employers may also seek to minimize the chances that algorithmic decision-making tools will assign poor ratings to individuals who are able to perform the essential functions of the job, with a reasonable accommodation if one is legally required. Promising practices include:
- Ensuring that the algorithmic decision-making tools only measure abilities or qualifications that are truly necessary for the job—even for people who are entitled to a reasonable on-the-job accommodation.
- Ensuring that necessary abilities or qualifications are measured directly, rather than by way of characteristics or scores correlated with those abilities or qualifications.
Before purchasing an algorithmic decision-making tool, an employer should ask the vendor to confirm the tool does not ask job applicants or employees questions that are likely to elicit information about a disability or seek information about an individual’s physical or mental impairments or health unless such inquiries are related to a request for reasonable accommodation. (The ADA permits an employer to request reasonable medical documentation in support of a request for reasonable accommodation that is received prior to a conditional offer of employment, when necessary, if the requested accommodation is needed to help the individual complete the job application process).
