Public Comments - Proposed Rules Related to Automated Employment Decision Tools

Comments Received by the Department of

Consumer and Worker Protection on

Proposed Rules related to Automated Employment Decision Tools

IMPORTANT: The information in this document is made available solely to inform the public about

comments submitted to the agency during a rulemaking proceeding and is not intended to be used for

any other purpose

Dear Sir / Madam –

On October 24, 2022, the Department of Consumer and Worker Protection of the City of New York will hold a

public hearing to discuss and hear comment on the aforementioned ordinance. We request that this ordinance

be written to more clearly articulate its true spirit, namely, to reduce or eliminate the use of artificial

intelligence and machine learning in selection processes. The following paragraphs share our rationale for this

request.

Criteria Corp. is a leader in pre-employment assessment (e.g., online tests, structured interviewing), with more

than 4,500 clients across more than 60 countries. Since our inception in 2006, we have delivered more than 40

million assessments to help thousands of organizations make better hiring decisions, enhancing diversity among

their workforce, and improving bottom line results. Criteria was founded on the belief that all job candidates

should have an equal opportunity to demonstrate their capabilities in a fair and unbiased manner, regardless of

their background or socioeconomic status. We use assessments to achieve this goal.

Criteria does NOT use artificial intelligence or machine learning in our assessments. All of Criteria’s assessments

are based in science, have a logical rationale for their use in selection for a given role, and are extensively proven

to predict job performance. For example, we know that there are more than 22,000 studies across more than 5

million participants, demonstrating that smarter employees tend to perform their jobs better, so Criteria has

assessments to measure candidate intelligence. There are countless studies demonstrating that employees who

work harder tend to perform their jobs better, so Criteria has assessments that measure candidates’ propensity to

try their best and work hard on a given job. There are many studies demonstrating that specific characteristics,

such as sales ability, customer service orientation, and empathy, contribute to success in sales, service, and

leadership jobs, respectively, so Criteria has assessments that measure these characteristics.

In addition to the published literature, Criteria has conducted hundreds of studies across our client base

demonstrating that our assessments predict job performance, and do so in a fair and unbiased manner. In essence,

our assessments empower job candidates to demonstrate their abilities in a manner that transcends their

backgrounds and socioeconomic status, as well as their gender, ethnicity, age, nation of origin, and other

characteristics that are irrelevant to job performance. The use of our assessments creates a fairer selection process

by removing bias.

Given that Criteria does NOT use artificial intelligence or machine learning in our assessments, we understand

that we are not impacted by this upcoming New York City law. However, the law as it is currently written is very

brief, and subsequently very broad, such that one could misconstrue our assessments to be impacted.

For example, given the brevity and subsequent breadth of the current law, if a company in New York City wanted

to hire an Accountant, and they used a math test as part of the hiring process, they could potentially be fined

based on this law. If a company in NYC wanted to hire an HR Manager who is tolerant and empathetic, and they

assessed job candidates for that position with an online test that measured tolerance and empathy, they could

potentially be fined under this law. If a company in NYC wanted to hire a leader who is strong in integrity, they

could potentially be fined if they used an assessment to hire that person.

In each of these examples, there is an a-priori rationale for the use of the test content (math for accounting;

tolerance and empathy for HR; integrity for leadership). However, given that all online tests measuring these

characteristics use an algorithm (even an algorithm as simple as just adding the total number of math questions

answered correctly), they could be construed by some as falling under this law.

We therefore ask that the law be written to more clearly articulate its true spirit, namely, to reduce or

eliminate the use of artificial intelligence and machine learning in selection processes. Note that the key

difference between a test using artificial intelligence and machine learning, versus a test that does not is based in

whether the test uses an a-priori rationale for what it is measuring. If the test has an a-priori rationale for what it

is measuring, then it is not using artificial intelligence and machine learning. We therefore ask that any

assessment using an a-priori rationale be excluded from this law.

Please also note that, given its efficiency, objectivity, and accuracy, testing cannot simply go away. There are far

too many job applicants for most companies to interview all of them, so testing needs to exist as an efficient,

accurate, and unbiased means for identifying candidates who are most likely to succeed on the job. The law as

currently written could possibly, and unintentionally, include virtually any test. It would be far better to target the

subset of tests that lack an a-priori rationale (e.g., artificial intelligence and machine learning based assessments

only) with this law.

Thank you for taking the time to read our request. We look forward to the upcoming public hearing.

Respectfully yours,

Brad Schneider, Ph.D.

SEC - COMMERCIAL IN CONFIDENCE

Brad Schneider, Ph.D.

Vice President, Strategic Consulting

mobile +1 (747) 800-1657

The following comments pertain to the proposed automated employment decision tools rules.

1. The proposed definition for “automated employment decision tool” could benefit from further explanation

and illustrations, particularly with regard to the phrase “or to use a simplified output as one of a set of

criteria where the output is weighted more than any other criteria in the set.” What about a situation where a

candidate score is generated that is based 90% on structured panel interview ratings (numerical ratings

averaged) and 10% on a tool that uses machine learning? Or a score that is based 90% on a job knowledge

exam (multiple choice test score) and 10% on a tool that uses machine learning?

2. The proposed bias audit is based solely on the generation of impact ratios, and thus equates bias with score

or categorization differences by group. As discussed in Chapter 3 of the 2014 Standards for Educational and

Psychological Testing, measurement bias, and more broadly fairness, are complex concepts. These

concepts cannot be boiled down to a simple impact ratio. Further, the validity of the scores or

categorizations must be considered to evaluate bias. The proposed bias audit process is therefore likely to

yield oversimplified, inaccurate and misleading results.

3. How is it envisioned that the bias audit would be handled when a new automated decision tool is initially

implemented? If it is a new tool, obviously the user would have no data to compute impact ratios. Would the

expectation be that the bias audit as defined would be performed after 1 year of use?

Joseph D. Abraham, Ph.D.

Vice President, Assessment Solutions

O: 01 818 847 6180, Ext. 7306 | M: 01 918 404 0050

Time zone: Central Time

Talogy.com | LinkedIn

This electronic mail, and any attachments, are intended solely for the person(s) to whom it has been sent and may contain information which is proprietary, business-

confidential and/or privileged material. If the reader of this message is not the intended recipient, or you have received this message in error, you are hereby formally

notified that any unauthorized review, use, disclosure, dissemination or reproduction of this message, or any of its contents, in whole or in part, is strictly prohibited.

Please immediately reply to sender, delete, and permanently and irretrievably destroy any copies and attachments of this message. Neither Talogy, nor its

subsidiaries, warrant that any electronic mail is secure or free of any viruses, and therefore does not accept liability for any damage caused by any virus transmitted

by any electronic mail.

October 24, 2022

VIA E-MAIL: Rulecom[email protected]

City of New York

Department of Consumer and Worker Protection

42 Broadway

Manhattan, New York 10004

Re: Letter of Comment on the DCWP Clarification for NYC Local Law 144

Dear Sir/Madam:

The Institute for Workplace Equality (“IWE” or “The Institute”) submits the following comments in response to

the New York City (“NYC” or the “City”) Department of Consumer and Worker Protection’s (“DCWP” or the

“Department”) invitation. The Department’s Notice of Proposed Rules is seeking to clarify the requirements set

forth by NYC’s Local Law 144 that will regulate the use of automated employment decision tools (“AEDT”)

wherein hiring decisions are made or substantially assisted by algorithmically-driven mechanisms.

Background on The Institute for Workplace Equality

The Institute is a national non-profit employer association based in Washington, D.C. The Institute’s mission

includes the education of federal contractors as to their affirmative action, diversity, and equal employment

opportunity responsibilities. Members of The Institute are senior corporate leaders in EEO compliance,

compensation, legal and staffing functions representing many of the nation’s largest and most sophisticated

federal contractors.

The Institute recognizes the responsibility of all employers to create a nondiscriminatory workplace. To that end,

NYC’s DCWP has an important role in enforcement efforts related to Local Law 144 and additional rules to

clarify the requirements for compliance. This is critical to ensuring that employers understand their requirements

and can effectively comply with the law beginning January 1, 2023.

I. Overview of Proposed Rules for Local Law 144

The Institute applauds the DCWP’s efforts to clarify Local Law 144. Enhanced clarity established by clearly

defining terms and requirements allow employers to ensure they are

compliant with this law when it goes into effect. Our members are encouraged by various clarifications in the

proposed rules so far.

However, there are some areas where the proposed rules fall short of providing clarifying guidance and/or deviate

from standards and accepted practices related to evaluating disparate impact. Additionally, there are areas in

which inconsistencies raise questions that require further clarification to allow employers and auditors to

understand how to properly comply with Local Law 144. The remainder of these comments will focus on these

areas in which we encourage the DCWP to consider further revising and clarifying the provided definitions and

requirements.

II. Who is eligible to conduct a bias audit?

In §5-300, the DCWP defines an independent auditor as: “… a person or group that is not involved in using or

developing an AEDT that is responsible for conducting a bias audit of such AEDT.” This definition suggests that

neither an employer using an AEDT nor a vendor developing an AEDT will suffice as an independent auditor.

However, in the first example of § 5-301, the statement is made that:

The employer asks the vendor for a bias audit. The vendor uses historical data it has

collected from employers on applicants selected for each category to conduct a bias

audit…

Understanding that this example is describing a situation in which the employer is considering the use of an

AEDT (as opposed to an employer having already used an AEDT) such that no employer-specific data are

available and historical vendor data may be required, the phrasing suggests that a vendor is conducting the bias

audit of their own AEDT for an employer.

It is important to distinguish between who is conducting the bias audit and who is providing the data upon which

the bias audit is based. There will most certainly be use cases, such as the one outlined in the DCWP’s provided

example, wherein an AEDT has not yet been deployed for an employer, and a vendor may need to provide

relevant data to an independent auditor to be used in conducting the bias audit analyses. However, a vendor

providing data for a bias audit and a vendor being eligible to serve as the independent auditor of their own AEDT

and conducting the bias audit are quite different. As such, we request the DCWP clarify this issue in the

proposed rules to reflect their intent. If the definition of independent auditor precludes vendors from performing

bias audits, then the example should clarify that the vendor is providing data for the bias audit to an independent

auditor, but not performing audits of their own tools.

III. What data should a bias audit be based on for different use cases?

The proposed rules do not make it clear whether employers are required to use their specific data or whether

cross-client, aggregate data from a vendor might suffice. We recognize that this answer may depend on the

specific circumstances. In some situations, an employer will have their specific data available to use (e.g., post

deployment of an AEDT, pilot studies with an AEDT), while other situations may occur where no employer-

specific data are available (e.g., prior to deployment of an AEDT). In situations where employer-specific data are

not available, there will be some circumstances in which historical, archival data from other sources may be

available, and some in which no historical, archival data are available (e.g., custom-built AEDT pre-deployment)

to estimate the potential bias of an AEDT.

We request the DCWP provide clarification for three typical use cases regarding the data that should be used for

the purpose of conducting bias audit analyses. Specifically, we recommend that the DCWP address the following

use cases and provide guidance in the proposed rules.

1. Implementing an AEDT where sufficient employer data are available: This scenario aligns well to

the Uniform Guidelines and should most closely mirror the reality of operational use of the AEDT.

When should employer data be analyzed for the bias audit?

2. Implementing an AEDT where sufficient employer data are not available, but sufficient cross-

employer vendor data are available: This is similar to a longstanding practice by vendors to use

aggregated data to estimate how an assessment would be expected to function, absent employer-

specific data. When should cross-client, aggregate vendor data be analyzed for the bias audit?

3. Implementing an AEDT where no existing data are available: How should employers proceed

when seeking to deploy a custom-built AEDT, for which no prior data exists? Are there options

available for auditors to evaluate bias via a proxy method (e.g., another similar type of AEDT,

research literature related to a type of AEDT), or will the City grant a grace period for this

particular use case?

Providing clarification on data eligible for analysis in these situations would be beneficial to employers and

auditors to ensure that the bias analyses conducted are compliant with Local Law 144.

Additionally, DCWP should seek feedback as to whether the specific language used in the examples in the

proposed rules are consistent with the remainder of the information contained in the proposed rules and with the

intention of the City.

IV. Are there parameters that define data to be used in bias audit analyses?

After determining the source of data that are eligible for use in different situations, it is important to consider

relevant parameters to conducting the bias audit. To build consistency in what employers are expecting, what

auditors are delivering, and what the City is evaluating for compliance, the City would be well served by directly

addressing the following parameters:

1. Candidate location: Should bias audit analyses restrict the candidate pool to include only NYC

residents, or may a larger candidate pool be considered (e.g., state, region, national)?

2. Relevance of data when employer-specific data cannot be used: In some situations, AEDTs may

be used across a very wide range of jobs (e.g., hourly workers, sales professionals, managers,

executives) or in very different settings (e.g., finance, manufacturing, customer service). Should

bias audit analyses be restricted those jobs and settings that more closely represent the jobs for

which the employer intends to use the AEDT?

V. The definition of impact ratio

We applaud the DCWP for providing more clarity around the bias audit analyses that are expected. The impact

ratio is defined in § 5-300 as:

[E]ither (1) the selection rate for a category divided by the selection rate of the most

selected category or (2) the average score of all individuals in a category divided by

the average score of individuals in the highest scoring category.

Impact Ratio = Selection rate for a category

Selection rate of the most selected category

Impact Ratio = Average score of individuals in a category

Average score of individuals in the highest scoring category

The first equation is one of the well-recognized, practical significance metrics for evaluating selection rate

differences between two different demographic groups. This metric is consistent with the “four-fifths rule” as

described in the Uniform Guidelines on Employee Selection Procedures

(“Uniform Guidelines” Section 4D) and

their Interpretation and Clarification (Questions and Answers) document

(“Q&A” Question 12).

The second equation, however, is not a metric that is used in practice based on our experience. It contains a

fundamental flaw that does not allow for meaningful conclusions about the difference between scores on an

AEDT for two different demographic groups because it does not account for variability in scores. This metric, as

defined, may lead to purposefully or

inadvertently illusory, misleading results. In simple terms, if a child comes home from school and tells you that

they scored an 85 on a test at school, your first question might be, “Is that good?” From the information provided,

you cannot tell—you lack relevant context. To know if this is a good score on the test, you need to know how

others did perform (or typically perform) on the test—you need to understand the distribution (variability) of

scores on the test.

That said, we very much appreciate that the DCWP has recognized the need for a bias indicator to handle the

situation wherein an AEDT is not being used with a strict threshold (e.g., cut-score, band, categorization) that

allows for computing a selection rate, but instead scores are generated and are being directly used as a factor in

determining which candidates proceed in a hiring process. This is a very common use case, and we agree that it

would be best if a consistent method were used across auditors.

To that end, we submit that there are multiple, professionally-accepted, practical significance measures that may

be used for effectively evaluating the difference between two “average scores” on an AEDT, and that the DCWP

consider adopting one of these. For example, one such statistic that is commonly used is the d-statistic.

345

simple terms, this metric allows one to describe the difference in magnitude of two average scores, regardless of

the distribution of scores, in a consistent, meaningful, and interpretable way.

Uniform guidelines on employee selection procedures, Equal Employment Opportunity Commission, Office of Personnel

Management, Department of Justice, Department of Labor, and Department of Treasury, 43 Fed. Reg. 38290-38315 (August 25,

1978). See 29 CFR Part 1607 (1978) available at https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607 and 42 CFR Part 60-3 (1978)

available at https://www.ecfr.gov/current/title-41/subtitle-B/chapter-60/part-60-3?toc=1.

Adoption of Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures;

Equal Employment Opportunity Commission, Office of Personnel Management, Department of Justice, Department of Labor, and Department of

Treasury (1979). 44 Fed.Reg. 11996-12009 (March 2, 1979); available at https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-

provide-common-interpretation-uniform-guidelines.

Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.

The d-statistic is sometimes referred to as an effect size difference or a standardized mean difference.

This represents a pooled standard deviation which is commonly used for this statistic. This provides a more accurate representation of the overall

standard deviation for the pairwise comparison of group 1 and group 2 averages.

To compare this to the second impact ratio metric described in the proposed rules, consider an example of three

different AEDTs where the average score of two groups is being compared. Then consider how the interpretation

of results is much different once you have information about the distribution of scores (i.e., the variance or

standard deviation). The second impact ratio defined in the proposed regulations is represented by the “Averages

Ratio” column, the variability of scores in the distribution is represented by the “Standard Deviation” column,

and the result of calculating the standardized difference in average scores is represented by the “d-statistic”

column.

Scenario Average

Group 1

Average

Group 2

Averages

Ratio

Standard

Deviation

d-statistic

AEDT 1 50 44 .88 24.0 0.25

AEDT 2 50 44 .88 12.0 0.50

AEDT 3 50 44 .88 6.0 1.00

In all three cases, the Averages Ratio appears to be the same—one could draw the same conclusion about the

difference in demographic group scores. However, when considering the distribution of scores with a statistic

such as the d-statistic, it is clear that the same conclusion should not be drawn. With AEDT 1, the difference is

relatively small because there is a large

amount of variation in scores; with AEDT3, the difference is large as there is little variation in scores. The

interpretation of bias for these two AEDTs would be quite different.

It should be clear from this example that the currently-defined second impact ratio is not very meaningful and

will be unlikely to yield interpretable results. Given this, we request that the DCWP define a practical

significance metric commonly used or professionally-accepted that will be more appropriate for use in comparing

the average score of two groups, of which the dstatistic is but one example.

VI. Parameters to clarify related to calculations of an impact ratio

After defining the impact ratio, there are practical parameters to a bias audit analysis that are important to

consider. These parameters are related to the stability of results and the confidence one might have in drawing

conclusions based on the analyses conducted. We recommend the DCWP consider addressing some key

parameters in the proposed rules or indicate that there is allowance for reasonable professional judgment by the

independent auditor.

Note that some of these parameters may be informed by the Uniform Guidelines and their Q&A document since

the proposed rules for Local Law 144 state that:

These calculations are consistent with Section 1607.4 of the EEOC Uniform Guidelines on Employee

Selection Procedures. See 29 CFR §1607.4.

1. Sample size required for a pairwise comparison: Small sample sizes are especially problematic for

impact ratio analyses. Evidence based on small numbers is considered in the Uniform Guidelines (4D)

and their Q&A document (e.g., Question 20, Question 21). For example, in Question 20 they provide

an example whereby an impact ratio based on a pool of 20 males and 10 females, where 3 males and 1

female are selected, yielded a conclusion that “the number of selections is too small to warrant a

determination of adverse impact.” In Question 21, they provide an example in which impact ratio

results would change appreciably with the shift of a single individual, noting that this indicated results

that were unstable. Given the nature of impact ratios, this instability at small sample sizes is to be

expected—the influence of a decision made about a single individual may unduly effect the statistical

conclusions.

We request the DCWP directly acknowledge the relevance of the federal guidance, directly specify

guidance on minimum sample sizes to warrant analyses, or state that this is subject to independent

auditor discretion.

2. Sample size required for a demographic group to be evaluated: We would appreciate the DCWP

confirming that the guidance set forth by the Uniform Guidelines Section

15A-2

and Question 16 of their Q&A document, indicating that groups which

29 CFR §1615(a)(2); 42 CFR § 60-3.15(A)(2).

comprise less than 2% of the pool should be excluded from analysis, should be applied when

conducting Local Law 144 bias audits.

3. Interpretive guidelines for impact ratios: Interpretation of an impact ratio has become more nuanced

over the years. Impact ratio results can be interpreted in a binary framework based on the four-fifths

rule, or with a multi-categorical interpretation (e.g., OFCCP Q&A on Practical Significance, January

15, 2021

), or with more of a continuous consideration taking the statistic as a way to gauge the

potential for a tool to yield bias. Will the DCWP provide interpretation guidance, or will this be left to

the discretion of the independent auditors?

4. Which impact ratio to calculate: §5-301-b indicates that the averages ratio must be calculated when

candidates are scored. However, in the vast majority of cases, any candidate that has been assigned to

a classification or band has been given a score upon which the assignment is based. As such, the

proposed rules read that a classification or banding approach will require both be calculated.

Given the definition of impact ratio in §5-300, which clearly indicates (via the “OR”) that the intent is

to calculate one metric (not both), it would behoove all for the DCWP to align language in the

proposed rules to reflect that only one metric should be calculated and that this should be the one that

most closely mirrors the operational implementation of the AEDT.

VII. Which pairwise comparisons are required for analysis?

The tables in the two examples in the proposed rules imply intersectional analyses need to be conducted and

reported (i.e., the combination of race/ethnicity and sex). While intersectional analyses can be conducted, in most

cases the primary interest is in evaluating race/ethnicity and sex separately, with intersectional analyses

considered as supplemental at most.

This is another situation in which it would behoove all involved for some clarification on the pairwise

comparisons that are intended to be evaluated in a bias audit for Local Law 144. Should auditors analyze

race/ethnicity and sex separately only, analyze intersectional combinations only, or do both (yielding three tables

per analysis strata: 1 race/ethnicity, 1 sex, 1 intersectional).

https://www.dol.gov/agencies/ofccp/faqs/practical-significance

VIII. Effective date for Local Law 144

NYC’s Local Law 144 is slated to go live on January 1, 2023. However, there is still significant confusion related

to the specifics for employers coming into compliance with this law. Further, as noted here, there are some

additional clarifications that are necessary and will impact the ability for employers and auditors to comply with

what is required. Finally, with the

rescheduling of the October 24 public hearing to a later date, both the feedback to the DCWP, as well as any

revisions to the proposed rules based on this feedback, will be further delayed. We respectfully request that the

City immediately consider a six month delay to the effective date. In the alternative, the City could proceed with

a soft launch in which there is a grace period for employers to comply, so long as an independent auditor is

identified and the work is scheduled by January 1, with a requirement to have the work completed before the year

concludes.

Conclusions

Thank you in advance for your consideration of The Institute’s comments. We are happy to provide any

additional information you may need or to answer any questions you may have.

Best wishes,

Barbara L. Kelly

The Institute for Workplace Equality Director

BSA | The Software Alliance Comments on the New York City Department of

Consumer and Worker Protection’s Proposed Rules Implementing Local Law 144 of

2021 Regarding Automated Employment Decision Tools

October 24, 2022

BSA | The Software Alliance appreciates the opportunity to submit comments on the proposed regulations

implementing Local Law 144 of 2021—New York City’s ordinance on automated employment decision tools

(“Ordinance”). BSA is the leading advocate for the global software industry before governments and in the

international marketplace. Our members are enterprise software companies that create the technology products

and services that power other businesses.

They provide trusted tools that help other businesses innovate and

grow, including cloud storage services, customer relationship management software, human resources

management programs, identity management services, and collaboration software.

BSA members are on the leading edge of providing businesses — in every sector of the economy — with

innovative software services, including Artificial Intelligence (AI). As leaders in the development of enterprise AI

systems, BSA members have unique insights into the technology’s tremendous potential to spur digital

transformation and the policies that can best support the responsible use of AI.

While the adoption of AI can unquestionably be a force for good, we also recognize the significant risks to society

if this technology is not developed and deployed responsibly. BSA agrees that when AI is used in ways that could

unlawfully discriminate, the public should be assured that such systems have been thoroughly vetted to identify

and mitigate risks associated with unintended bias.

A. BSA’s Framework to Build Trust in AI

This has been an area of particular focus for BSA and our member companies over the last several years. In June

2021, BSA released Confronting Bias: BSA’s Framework to Build Trust in AI, which sets forth a risk

management approach for confronting concerns about bias.

The Framework was informed by leading AI

developers and outlines a lifecycle-based approach for performing impact assessments to identify risks of AI bias

and corresponding best practices for mitigating those risks. AI is used in so many different contexts that only a

flexible, risk management approach will be successful. There are a variety of AI development and deployment

models, and the Framework recognizes that the appropriate allocation of risk management responsibilities will

BSA’s members include: Adobe, Alteryx, Atlassian, Autodesk, Bentley Systems, Box, Cisco,

CNC/Mastercam, CrowdStrike, Databricks, DocuSign, Dropbox, Graphisoft, IBM, Informatica, Intel,

Kyndryl, MathWorks, Microsoft, Okta, Oracle, Prokon, PTC, Salesforce, SAP, ServiceNow, Shopify Inc., Siemens Industry Software Inc., Splunk,

Trend Micro, Trimble Solutions Corporation, TriNet, Twilio, Unity Technologies, Inc., Workday, Zendesk, and Zoom Video Communications, Inc.

Confronting Bias: BSA’s Framework to Build Trust in AI, available at https://ai.bsa.org/confrontingbias-bsas-framework-to-build-trust-in-ai.

vary depending on the type of system, including who develops the algorithm, trains the model, and ultimately

deploys the system.

BSA supports the overarching goal of the proposed regulations, which is to ensure high-risk uses of AI are subject

to safeguards. One crucial safeguard that promotes responsible uses of AI systems is ensuring that companies that

develop or use high-risk AI systems establish a comprehensive approach for performing impact assessments.

Impact assessments are widely used in a range of other fields—from environmental protection to data

protection—as an accountability mechanism that promotes trust by demonstrating that a system has been designed

in a manner that accounts for the potential risks it may pose to the public. The purpose of an impact assessment is

to establish organizational processes to guide the development and use of high-risk systems by requiring internal

stakeholders to identify the risks that a system may pose, quantify the degree of harm the system could generate,

and document any steps that have been taken to mitigate those risks to an acceptable level. By establishing a

process for personnel to document key design choices and their underlying rationale, impact assessments are an

important transparency and accountability mechanism.

B. The Department of Consumer and Worker Protection’s Proposed Regulations

The proposed regulations address the need for safeguards that promote responsible uses of AI by focusing on how

companies are to implement bias audits under the Ordinance. As the proposed regulations recognize, one critical

aspect of operationalizing such audits is identifying a set of individuals who can conduct them. Importantly, the

proposed regulations appropriately recognize that internal stakeholders who are not involved in the development

or use of an automated employment decision tool are competent to conduct a bias audit. Acknowledging the

independence of internal stakeholders will incentivize companies to implement multiple layers of independent

review, which will enhance trust in the use of these systems and create safeguards that function in practice.

We have recommendations, however, about three aspects of the proposed regulations:

First, we suggest deleting “or modify” in the definition of “automated employment decision tool” in Section

5-300. The definition refers to relying “solely” on simplified outputs “with no other factors considered” and use

of simplified outputs to “overrule” conclusions derived from other factors including human decision-making. As

illustrated by these descriptions, the intent of this definition appears to be identifying circumstances that could

“substantially assist or replace discretionary decision making.” This is a relatively high threshold. Outputs that do

not rise to this level, such as merely contributing to or modifying, should be excluded from the scope of this

defined term.

Second, we suggest adding a notation explaining that the examples in Section 5-301 for the bias audit are

not exhaustive of all scenarios. Along the same lines, we recommend revising the two examples in this section

to recognize a broader range of possible scenarios regarding the appropriate entity to conduct a bias audit. There

are myriad development and deployment models, and multiple stakeholders may play varying roles depending on

the circumstances. Specifically, we recommend offering an illustrative example without inadvertently suggesting

that the types of entities described in the example will always have the same roles.

Further, in the examples for both subsections (a) and (b) of this section, we recommend replacing “historical data”

with “test data” because there may not always be historical data if the organization has not previously used the

technology. In addition, historical data may not be available in many scenarios—such as when that data is subject

to privacy or security safeguards that prevent its dissemination. In addition, we recommend deleting “planned”

from the example in subsection (b), as it inserts ambiguity regarding the timing of the use of the automated

employment decision tool.

Third, we recommend omitting the requirement to publish the selection rates and impact ratios for all

categories and instead require a summary statement on adverse impact. Although Section 20-871(2) of the

Ordinance requires a “summary of the results” of the bias audit to be published, it does not call for the level of

specificity contemplated by the proposed regulations. Publishing the specific information contemplated by the

proposed regulations could inadvertently undermine the goals of the Ordinance. For example, it may discourage

applicants from groups that are selected less frequently from applying to an organization at all, hampering efforts

to attract a diverse workforce. Moreover, requiring the public disclosure of such specific information could

disincentivize companies from conducting thorough audits to avoid possible results that may not be optimal.

Accordingly, we recommend striking “the selection rates and impact ratios for all categories” in Section 5302 and

replacing it with “a statement on adverse impact.” We further recommend aligning the categories for the

selection rates and impact ratios with the EEOC’s approach for disparate impact testing.

We thank you for the opportunity to comment on the proposed regulations and look forward to serving as a

resource as you finalize the proposed regulations.

October 24, 2022

To: New York City Department of Consumer and Worker Protection

42 Broadway

New York, NY 10004

Re: Proposed Rules to Implement Local Law 144 of 2021 on Automated Employment Decision Tools

The Center for Democracy & Technology (CDT) respectfully submits these comments on the

Department of Consumer and Worker Protection’s proposed rules relating to Local Law 144 of 2021 (“LL 144”),

the ordinance that the New York City Council passed last fall relating to employers’ notice and audit

requirements for the use of Automated Employment Decision Tools (AEDTs). CDT is a nonprofit, nonpartisan

501(c)(3) organization based in D.C. that advocates for stronger civil rights protections in the digital age,

including algorithmic tools that can affect workers’ access to employment.

CDT supports laws and regulations that ensure that AEDTs are nondiscriminatory and job-related, include proper

candidate notice, and are subjected to adequate auditing, oversight, and accountability. CDT appreciates the

Council’s efforts to establish some level of accountability for AEDTs through the passage of LL 144, and

likewise appreciates the

Department’s efforts to bring clarity to that ordinance through its proposed rules. Nevertheless, as CDT explained

after the ordinance passed, we do not believe the ordinance goes far enough in ensuring that AEDTs are tested for

potential discriminatory effects, that candidates receive proper notice of the manner in which AEDTs assess them,

or that adequate enforcement and remedies are available.

Unfortunately, several of the proposed rules would further diminish the ordinance’s effectiveness by narrowing its

scope and application in ways that are not supported by the ordinance’s text. Our comments discuss the ways in

which we believe the draft rules can be strengthened to ensure that the City’s workers are protected to the

maximum degree possible under the ordinance.

Proposed interpretation of “Automated Employment Decision Tool” is narrower than the statutory

definition

The statutory definition of “automated employment decision tool” covers all automated processes that

“substantially assist or replace discretionary decision making” in employment decisions.

The proposed rules

interpret this term as requiring that the output of an AEDT be the “sole[]” factor in a decision, be “one of a set of

criteria where the output is weighted more than any other criterion in the set,” or “overrule or modify conclusions

derived from other factors.” This interpretation is unjustifiably narrower than the statutory definition, which

encompasses any system that “substantially assist[s]” human decision-making. The term “substantial” means “of

ample or considerable amount, quantity, size, etc”

as well as items that are “important” or “essential.”

Therefore, the plain meaning of the statutory definition would designate tools used in a broad range of contexts as

AEDTs.

By narrowing the definition to tools that are given sole or primary weight, the proposed rules create a loophole

companies could exploit. The rules do not require employers to provide meaningful transparency about how the

outputs of AEDTs are weighted and about other factors considered to assess the same candidate. For an AEDT

whose output is “one of a set of criteria,” an employer could easily design a selection process that circumvents the

Matthew Scherer and Ridhi Shetty, NY City Council Rams Through Once-Promising but Deeply Flawed Bill on AI Hiring Tools, Center for

Democracy & Tech. (Nov. 12, 2021), https://cdt.org/insights/ny-city-council-rams-through-once-promising-but-deeply-flawed-bill-on-ai-hiring-tools/.

N.Y.C. Admin. Code §20-870.

Dictionary.com, substantial, based on the Random House Unabridged Dictionary,

https://www.dictionary.com/browse/substantial (accessed October 11, 2022).

Id.

ordinance or argue that the AEDT does not meet the rules’ narrow definition of “substantially assist.” For

instance:

● An employer could rely on multiple distinct AEDTs and give all of their outputs the same weight;

under the rules’ definition, none would then qualify as AEDTs.

● An employer could rely on multiple AEDTs whose outputs are weighted differently, so only the one

given the most weight would be subject to the ordinance’s requirements.

● An employer could give an AEDT’s output the same weight as the outputs of non-automated tools or

methods used in the overall process, and such an AEDT would fall outside the audit requirements

under the rules.

The proposed rules could also lead to inconsistent results--an AEDT that is given 40% weight in a hiring decision

might not qualify (if another factor is weighted the same or slightly more), while one that is only given 15%

weight might qualify (if there are numerous other factors with lesser weight also considered).

We urge the Department to either leave the term undefined, as the word “substantial” is commonly understood

and can be given its plain and ordinary meaning, or else to define it as including tools whose output is “an

important or significant factor in an employment decision.”

“Candidate for employment” should include all workers subjected to an AEDT

The ordinance defines “employment decision” to mean “to screen candidates for employment or employees for

promotion in the city.” This definition is already too narrow as it only covers decisions to hire or promote.

This

excludes decisions regarding “terms, conditions, and privileges of employment” that were previously covered in

the legislative text when it was originally introduced.

The proposed rules would unduly restrict this definition even further by limiting what it means to “screen” a

“candidate for employment.” The proposed rules specify that to “screen” is to determine “whether someone

should be selected or advanced in the hiring or promotion process” and that a candidate is only someone “who has

applied for a specific employment position.” Those definitions are inconsistent with the text of the ordinance;

while the ordinance’s notice requirement only applies to “a candidate who has applied for a position for an

employment decision,” the bias audit requirement applies more broadly to tools used by employers to “screen a

candidate for an employment decision.” The rules’ definition thus inserts language that is not consistent with the

term’s use in the text of the ordinance.

By narrowing the ordinance’s scope to the screening of candidates who actively apply for a job, the proposed

rules exclude several categories of automated tools, such as those that target job advertisements or search resume

databases or social media to recruit potential candidates. This would exclude a significant number of workers

from the ordinance’s protection because such tools increasingly determine the candidate pool for jobs in the city.

The definition of “screen” is also seemingly narrower than the definition of discriminatory practices under the

current City Administrative Code, which states that it is unlawful to “represent that any employment or

position

is not available when in fact it is available.” N.Y.C. Admin. Code § 8-107(a)(1).

The rules instead should align more closely with the City’s civil rights laws by clarifying that to “screen” means

to “make a determination about whether someone should be selected or advanced in the hiring or promotion

process, or whether to represent that any employment or job position is available, on the basis of characteristics

protected under N.Y.C. Admin. Code § 8-107.” The rules should define “candidate for employment” as “a person

N.Y.C. Admin. Code §20-870.

Int. No. 1894.

Note that even among workers who do submit an application, the proposed rules’ requirement that a worker submit information “in the format

required” by the employer could give employers an opportunity to exclude workers from the bill’s notice requirements (or from the adverse impact

calculations in a bias audit) on a technicality.

RCNY §5-249 (emphasis added).

who applies for a position or who is subjected to an employment decision.” This will ensure that all people

screened or assessed by an AEDT--and thus all people whose employment prospects the AEDT affects--are

covered by the ordinance’s bias audit requirements.

Definition of “employment agency” relies on narrow language and should be clarified

The proposed rules use the definition of “employment agency” given in § 5-249 of the Department's regulations,

which only says that an “employment agency” includes “all persons who, for a fee, render vocational guidance or

counseling services, and who directly or indirectly represent” that they do one of the enumerated functions that

can help people apply for or secure jobs.

The same provision states that “vocational guidance or counseling

services” means “services which consist of one or more oral presentations” and perform other enumerated

functions for people seeking employment.

Since the proposed rules only apply to people who have applied for

specific job positions, the proposed rules’ reference to § 5-249 is only relevant to entities that provide oral

presentations to jobseekers and that represent that they can secure job positions for the jobseekers who have

already applied to those positions.

Given the increasing prevalence of employer service providers in the labor market, the definition of “employment

agency” should be broad enough to subject any person using AEDTs to try to procure employees for employers or

job opportunities for workers to the bill’s notice and bias audit requirements. New York State’s full definition of

“employment agency” captures employer service providers because it applies to “any person who, for a fee,

procures or attempts to procure (i) employment or engagements for someone seeking employment or

engagements, or (ii) employees for employers seeking the services of employees.”

Similarly, NYC's anti-

discrimination laws define the term as “any person undertaking to procure employees or opportunities to work.”

Both of these definitions would apply to entities that the proposed rules aim to hold accountable. Therefore, the

final rules should more directly refer to the broader definition in the State’s law and the City’s anti-discrimination

laws.

The requirements for “independent auditors” should be strengthened to ensure truly independent audits

The proposed definition of “independent auditor" would ensure some semblance of independence by requiring an

auditor be “a person or group that is not involved in using or developing” the AEDT for which they perform a

bias audit. This does not go far enough, however, in ensuring true independence. An independent auditor should

be required to be free of financial or personal conflicts of interest. In addition, the auditor should have a duty to

accurately report the results of all tests and analyses conducted during the course of the audit; otherwise, a vendor

may be able to pressure an auditor to withhold or modify audit results that are unfavorable to the vendor.

Consequently, we recommend that “independent auditor” be defined as “a person or group that is responsible for

conducting a bias audit of an AEDT and that:

● Is not involved in using or developing such AEDT;

● Certifies in the audit report that all results from the audit have been reported and are complete and

accurate; and

Id.

N.Y. Gen. Bus. Law § 171 (emphasis added).

N.Y.C. Admin. Code § 8-102.

● Has no financial, personal, or other conflict of interest that would compromise or call into question the

auditor’s duty or ability to design and conduct the audit to measure impact effectively and to

completely and accurately report the audit results.”

Notice should be communicated directly to candidates

The ordinance requires an employer to give candidates notice that an AEDT will be used to assess them “no less

than ten business days before such use and allow a candidate to request an alternative selection process or

accommodation.”

The rules permit an employer to satisfy this notice requirement simply by posting such a

notice “on the careers or jobs section of its website in a clear and conspicuous manner.” But the mere posting of

such information does not satisfy the notice requirement contemplated by the ordinance.

As the ordinance text indicates, a key purpose of this requirement is to allow candidates who may need to request

an alternative assessment method or accommodation an opportunity to do so.

The information required by § 20-

871(b)(2) regarding the “job qualifications and characteristics” the AEDT will measure is, relatedly, essential for

a candidate to determine whether they may need to make such a request. However, these requirements to notify

candidates about the use of an AEDT and qualifications and characteristics it will measure are not enough to

ensure candidates have the opportunity to request accommodations or alternative assessment methods.

First, neither the ordinance nor the proposed rules define “job qualifications and characteristics.” This leaves

employers with the discretion to provide vague or limited information about the characteristics or qualifications

that an AEDT will assess. Posting such information on a website does not address the information gap candidates

may encounter. Rather, notice should describe to candidates all criteria that the AEDT will evaluate, the purpose

of evaluating that criteria, and the data used to measure those criteria.

Second, by allowing employers to satisfy their “notice” obligation by posting the information on a website, the

ordinance effectively places the onus on workers to find out that they are being subject to an AEDT and how the

AEDT will be used. This exacerbates a flaw in the ordinance itself, which does not require employers to

proactively provide candidates with notice about the type of data an AEDT will collect, the sources of such data,

or the employer’s data retention policy. Rather, it requires employers who do not include this information on their

websites to provide it to candidates upon written request. The proposed rules simply affirm this, only adding a

requirement to post instructions on employers’ websites on how to make such written requests--again, placing the

onus on candidates to find the necessary information. This is not “notice” in any meaningful sense.

Third, neither the ordinance nor the proposed rules require employers to ensure that websites where notices are

provided are easily accessible to candidates with disabilities or candidates without reliable Internet access.

Further, the proposed rules do not clarify that employers have a duty to fulfill requests for reasonable

accommodations or alternative assessment methods.

We recommend that the final rules:

● Affirm employers’ duty to provide accommodations, recognizing that the use of an AEDT does not relieve

employers of this responsibility.

● Require employers to proactively notify candidates that they will be assessed by an AEDT, with a

description of the qualifications and characteristics the AEDT will measure, types of data it will use, the

sources of the data, and employers’ data retention policies.

● Require employers to provide this notice through multiple channels that make the notice accessible to

candidates with different needs, rather than allowing employers to choose a single method of notice that

does not effectively give candidates the information they need to decide whether they should request

accommodations.

N.Y.C. Admin. Code § 20-871(b)(1).

Id.

Conclusion

While LL 144 recognizes the need for accountability in the use of AEDTs, the law’s requirements fall short by

merely clarifying employers’ existing obligations to report race-, ethnicity-, or sex-based disparate impact and

offering notice requirements that employers could easily circumvent. The proposed rules do not effectively

implement--much less strengthen--the law’s obligations. Instead, the proposed rules limit the law’s protections

further with narrowed definitions that would exclude several tools from the auditing and transparency

requirements. We urge the Department to ensure that final rules maximize the law’s requirements to more

effectively hold employers accountable for preventing discriminatory outcomes when they deploy AEDTs.

Respectfully submitted,

Matt Scherer Ridhi Shetty

Senior Policy Counsel for Workers’ Rights

Policy Counsel

Privacy & Data Project

Privacy & Data

Project

[email protected] [email protected]

October 24, 2022

Commissioner Vilda Vera Mayuga

NYC Department of Consumer and Worker Protection

Consumer Services Division

42 Broadway, 9th Floor

New York, NY 10004

Re: Comment on Proposed Rules

Dear Commissioner Mayuga:

The use of A.I. in the hiring and promoting process has been essential in helping streamline the review, outreach,

vetting, and onboarding process of potential employees. The recent data from the Labor Statistics for the New

York City Region indicates that "The city's seasonally adjusted unemployment rate was 6.6 percent in August

2022, up 0.6 percent from July.”

This increase in the unemployment rate is due to a "rise in the number of

people entering the labor force.

In a time when the region is seeing growth in the labor market, and people are

looking to enter the workforce, we feel it is essential that the City make sure that the rules implementing Int.

1894-2020 in relation to “automated employment decision tools,” are made in a thoughtful and balanced manner,

which will allow for the deployment of such tools to benefit the employer, employee, and/or independent

contractor to help streamline the process.

The U.S. Chamber of Commerce has long recognized that "fostering public trust and trustworthiness in A.I.

technologies is necessary to advance its responsible development, deployment, and use." For this reason, we at

the Chamber appreciate the opportunity to provide the following comment on the New York City Automated

Employment Decision Tools Regulation, which we believe will help provide more certainty to the framework.

Definition of Automated Employment Decision Tool , We urge you to revise the definition of “automated

employment decision tool” and, more specifically, the definition of the phrase “to substantially assist or replace

discretionary decision making.” First, we would ask that you strike the last phrase "or to use a simplified output to

overrule or modify conclusions derived

from other factors including human decision making." In its place, we ask that you add the

following sentence: "Automated employment decision tool,’ or ‘AEDT,’ does not include the

automated searching of resumes to identify candidate qualifications, including relevant skills or experience."

Furthermore, most employers do not use AEDT as the sole factor for determination of whom to employ but use it

in a more holistic approach as one of many factors in evaluating a candidate. Although an employer may deploy

an algorithm on every potential candidate, that does not mean that the output is always used by the employer.

While an AEDT may review

https://dol.ny.gov/labor-statistics-new-york-city-region

https://ebglawmy.sharepoint.com/personal/aforman_ebglaw_com/Document

and/or score each candidate, an employer may still empower the hiring manager with the discretion to determine

whom to interview and the amount of reliance and weight they put on the tool’s output. With hiring managers

potentially weighing their expertise and analysis more than the AEDT output, we ask for clarification on category

(2) (“to use a simplified output as one of a set of criteria where the output is weighted more than any other

criterion in the set”).

Finally, if you choose not to delete category (3) (“to use a simplified output to overrule or modify conclusions

derived from other factors including human decision making”), we ask that “or modify” be struck or clarification be

provided on what it means to “modify” a conclusion.

Bias Audit: The examples provided in subsections (a) and (b) are both prescriptive in who bears responsibility for

the bias audit (i.e., the employer/deployer or the vendor/developer) without accounting for the range of possible

scenarios. For this reason, we prefer that the examples be made clear that they aren't necessarily exhaustive of all

scenarios and remove the specificity of responsibility in each of the two examples, allowing for flexibility to

account for the range of scenarios.

We request the following changes to subsection (a):

• Revise the initial phrase to read: "Where an AEDT is used to screen a candidate or employee for an

employment decision, a bias audit required by § 20-871 of the Code

must, at a minimum"

• In the example, strike "historical data" and replace it with "test data."

We request the following changes to subsection (b):

• In the example, strike the word “planned” from the phrase “planned use of the AEDT.”

• Also, in the example, strike "historical data" and replace it with "test data."

Finally, both examples suggest that the bias audit should compare selection rates of not just gender and

race/ethnicity – the usual categories required to be compared under the Uniform Guidelines of Employee

Selection Procedures – but also on the intersectional categories of gender and race/ethnicity (e.g., Hispanic

Males, Non-Hispanic Female Whites, etc.). Data on these intersectional categories, however, typically is not

collected by employers or vendors, as applicants and employees are given the opportunity to separately self-

identify their gender and their race/ethnicity. Furthermore, many employers and vendors do not collect any

gender or race/ethnicity data on their applicants; please clarify how such employers and vendors should conduct

a bias audit in the circumstance in which they do not have any demographic data.

Published Results: We ask for the addition of the following italicized phrase in section (a) "Prior to the use of an

AEDT to screen a candidate or employee for an employment decision, employers and employment agencies in the

city must make the following publicly available on the careers or jobs section of their website in a clear and

conspicuous manner:"

We also suggest striking the phrase in subsection (a)(1) "the selection rates and impact ratios for all categories," and

replacing it with "a statement on adverse impact."

Definition of Screen in relation to Employment Decisions: The law states that "employment decision" means "to

screen candidates for employment or employees for promotion within the city." The proposed rule defines

"screen" as "to make a determination about whether someone should be selected or advanced in the hiring or

promotion process." We ask for clarity on how the use of some A.I. tools would fit within this definition.

Vendor Audits: The proposed rules contain an example in §5-301(a) that strongly implies that employers can rely

upon bias audits commissioned by vendors using historic applicant data collected by the vendor and not the

employer's own data. We ask that the rule explicitly state that this is permissible and satisfies the “bias audit”

requirement.

Frequency of Audits: The law states that an AEDT cannot be used unless a bias audit was "conducted no more

than one year prior to the use of such tool." It is not clear whether that language requires yearly bias audits of the

tool or if conducting one audit on a tool is sufficient unless or until the tool is replaced or materially modified.

The Statement of Basis and Purpose of the Proposed Rule states that a bias audit is required "within one year of

use of the tool" which implies that the audit may take place within the 12 months which follow implementation

of the tool. We understand this is not the intent, so this should be clarified.

Grace Period: While we understand the legislation requires the rules and regulations to go into effect by January

2023, the U.S. Chamber strongly encourages the Department to provide a grace period of at least twelve (12)

months to businesses and organizations as they prepare to implement the final rule.

Conclusion: We appreciate the opportunity to comment on the implementing rules. It is essential that these

regulations are implemented by New York City in a manner that does not impose overly broad requirements,

which in turn could create significant uncertainty regarding the use of automated tools in hiring. Potential

limitations of the use of technology for hiring purposes for businesses could lead to unnecessary barriers to

finding qualified candidates for a job; this is particularly challenging during periods when we see both labor

shortages and increases in the labor market, as businesses are put in a position where they receive more

resumes/applications than they have the capability to review, which inhibits their ability to identify potential

candidates. The use of automated employment decision tools is essential in helping streamline the hiring and

promotion process. Thank you for considering the aboveproposed changes to give the business community the

necessary certainty they will need. If you have questions, do not hesitate to contact Michael Richards at

[email protected]m.

Sincerely,

Tom Quaadman

Executive Vice President

Chamber Technology Engagement Center

U.S. Chamber of Commerce

Indeed’s Comments on Proposed Regulations to NYC Local Law No. 144

New York City Department of Consumer and Worker Protection

October 24, 2022

Indeed’s mission is to help people get jobs. We are committed to increasing fair hiring and removing barriers for

job seekers. In the New York City area, 5.5M job seekers use Indeed to search for work and 124,000 businesses

use Indeed to find talent (Indeed Hiring Lab).

In the next 10 years, Indeed is committed to helping 30M job seekers facing barriers get hired. In order to reach

our goal, we must ensure that algorithms increase fairness and inclusiveness in hiring. At Indeed, we monitor our

algorithms for unintended bias by establishing checkpoints set out by Indeed’s AI Ethics Committee, a team

within ESG composed of data scientists, software engineers, product managers, and more.

In partnership with the government, we leverage our tools and solutions to address the most pressing hiring

challenges in our community. In a recent effort, Indeed joined the White House call-to-action to address the

current shortage of education workers by committing to host a series of hiring events to help public schools hire

teachers, administrators, counselors, and other school staff.

Our goal is to help New Yorkers find jobs and the right regulation on AEDTs can minimize harm and maximize

fast, successful hires. We appreciate New York City’s leadership on this important issue.

Feedback on Proposed Regulation

We support the Proposed Regulations to LL No. 144 for providing necessary clarity and we’d like to raise a few

points for further clarification.

Under section 5-301 “Bias Audit”, the “Example” discloses 1) demographic categories that intersect (both race

and gender) and 2) sample size of candidates who applied and were selected.

1) Intersectional Demographic Categories: Reporting on multiple demographic categories within one group

could allow individuals to be identified (particularly with small sample sizes) and may raise privacy

concerns. Alternatively, data would be more anonymous if audits were conducted for race and gender

separately. We’d like clarification as to whether it is required to conduct selection rates and impact

ratio for each intersectional group.

2) Sample Size: Sample size metrics (number of applications, number of selections) can correlate to

meaningful financial information about company performance. If this information is disclosed to clients

and competitors, it could create financial concerns. We’d like to confirm that it is not required to

disclose sample size as part of the bias audits.

Thank you for leading on this new issue and we’d love to be a resource as you finalize your regulations. I look

forward to the opportunity to discuss our feedback and any other input you receive. Please feel free to reach out to

Alison Klein (aklein@indeed.com) with any further questions.

October 24, 2022

Vilda Vera Mayuga

Commissioner

New York City Department of Consumer and Worker Protection

42 Broadway

New York, NY 10004

By electronic delivery: LinkedIn Comments on the New York City Department of Consumer and

Worker Protection’s Proposed Rules Implementing Local Law 144 of 2021 Regarding Automated Employment

Decision Tools

Dear Commissioner Mayuga:

LinkedIn, a New York City (NYC) employer with over 1,700 NYC employees, appreciates the opportunity to

submit comments on the proposed rules issued by the NYC Department of Consumer and Worker Protection

(Department or DCWP) to implement Local Law 144 of 2021 regarding Automated Employment Decision Tools.

LinkedIn is the world’s largest professional network, whose vision is to provide economic opportunity for every

member of the global workforce. LinkedIn provides an online platform that allows our 850 million members

worldwide to create professional connections, access employment opportunities, and learn new skills. The

platform also provides tools for employers and recruiters to search for prospective job candidates.

As a company whose mission to connect workers lies at the intersection of talent and technology, LinkedIn not

only recognizes the opportunities that come with the use of automated technologies in the employment sector,

including those using artificial intelligence and/or machine learning, but also fully recognizes the importance of

thoughtful and responsible use of these technologies. In carrying out our mission to create economic opportunity

for every member of the global workforce, delivering fair and equitable experiences for each of our 850M

members is paramount to this work. As a result, we design our systems in ways that help us provide the right

protections, mitigate unintended consequences, and ultimately better serve our members, customers, and society.

We share the goals of the NYC City Council in seeking to ensure that workers are protected throughout the

recruiting and hiring process and that the right skilled candidates have an equitable opportunity to thrive in the

workforce. As such, we applaud the Department’s efforts in the proposed regulations to provide additional clarity

with regard to the scope of tools that are covered by Local Law 144, along with the Department’s recognition that

there are various legal and other considerations that are applicable during the early stages of recruiting and

sourcing of candidates. It is at this very point in the hiring process—searching for prospective candidates—where

we believe automated tools can many times serve an important function of increasing access to the job searching

process for potential candidates of all backgrounds.

We appreciate the Department’s attention to these important issues and the opportunity to provide feedback on the

proposed rules. Our goal is to ensure that any final regulations are clear, harmonized with existing employment

laws where appropriate, and avoid creating any unintended barriers to implementation. Please find our comments

below, including three general themes related to the proposed Definitions, the Bias Audit Process, and the Timing

of Notice requirements.

DEFINITIONS

“Automated Employment Decision Tool”: LinkedIn appreciates the improved clarity that the proposed

regulations provide with regard to the scope of tools regulated by Local Law 144. However, given the general

definition of automated employment decision tool (AEDT) in the proposed regulations focuses on a tool that

replaces or substantially assists human decision making, we are concerned that the third alternative definition is

unclear in that it refers simply to use of a simplified output to “modify” a human decision. In practice, when

simplified outputs from tools are used as data points as part of a holistic human review, it will be difficult for

employers to identify when and how this language may apply, resulting in inconsistent employer and candidate

experiences that do not align with the apparent intent of the law.

To aid in clarifying the definition of “automated employment decision tool” in the proposal, we would

recommend striking the phrase “or modify” from the final clause of the definition.

“Screen”: Ensuring internal consistency across definitions within the regulation, particularly where important

distinctions are made, is vital to achieve the regulation’s intended goals. We appreciate the Department’s

recognition, in its proposed definition of “candidate,” that there are different laws and policy considerations that

come into play during the sourcing/candidate discovery phase of recruiting. Notably, there is no “screening out”

of individuals happening at that early stage; rather, this is the stage at which recruiters and employers search for

prospective candidates to invite to submit applications to a publicly available position. In our experience, this is an

area of great opportunity for modern technologies to enable hiring companies and other employers to conduct

such searches with the specific goal of developing a diverse candidate pool. This may help employers advance

affirmative action obligations consistent with federal and state laws, where they exist, or simply create a broader

and more diversified pool of candidates for a role.

To increase clarity and consistency across the draft regulation’s definitions and advance the goals of the proposal,

we would recommend replacing “someone” with “a candidate or employee” in the definition of “screen.”

BIAS AUDIT PROCESS

Classifications: The way in which this section speaks to “classifying individuals into groups” may be confusing to

employers, causing different employers to apply the regulations differently. Typically, in the context of

employment decision-making audits, using a selection rate/impact ratio calculation suggests that there is a

positive outcome (an individual is hired or moves forward in process) and a negative outcome (an individual is

rejected or does not move forward) to be compared for the purpose of the calculations. If a tool is used in some

way to classify individuals into groups, it is entirely possible that there is no “advantaged”—or positive—group

but simply differences among groups.

For example, one could envision a tool that classifies resumes with significant sales experience as primarily sales

related and resumes with significant engineering experience as primarily engineering related, but without creating

any preference or advantage in the hiring or promotion process. Calculating selection rates and impact ratios

across such classifications would not then yield meaningful outcomes about any sort of screening or selection and

may create confusion and unnecessary burden.

We would recommend amending the language to clarify that the regulation applies where an AEDT “classifies

those individuals into groups in a way that prefers a group to move forward in a hiring or promotion process.” In

addition, we recommend replacing “individuals” with “candidates or employees” for the same reasons related to

providing clarity and consistency across the draft regulation identified in our comments on the definition of the

term “screen.”

Additionally, it would be helpful to insert language that clarifies that the included examples are only that—

illustrative examples of acceptable methods of bias audit—and not prescriptive of exactly who is responsible for

conducting audits or the specific data required to be audited in every case. In our reading of the proposed

regulations, this appears to be the intended meaning, but providing express acknowledgment of this intent would

improve understanding and enable better employer compliance.

Timing of the Audit: The plain language of the regulation, and the proposed regulations’ reference to the

distribution date, appear to contemplate that a bias audit must be conducted within a year of the AEDT being

implemented/placed into use. Nonetheless, questions remain about whether the language creates an obligation for

employers using an AEDT to perform an annual bias audit. We believe this is not the intent of Law 144 based on

1) the plain language of the law; 2) the fact that the alternative would imply some sort of rolling year calculation

where every single individual use of the AEDT would trigger the bias audit calculation (an untenable and

infeasible requirement); and 3) the fact that once an AEDT is in use as a screening procedure, it is already subject

to existing employment laws such as the Uniform Guidelines on Employee Selection Procedures (UGESP) and

general adverse impact discrimination law, which create the need for monitoring of ongoing impact. The proposed

regulations do appear to recognize this in various provisions of the draft (e.g., Example 2 in the Bias Audit section

and the incorporation of the distribution date definition into the disclosure section). For avoidance of doubt, we

suggest clarifying that the audit requirement is intended to be conducted in the year before placing an AEDT into

use.

To facilitate this clarification, we recommend modifying section 5-302(a) to clarify that the bias audit is required

“prior to implementing use of an AEDT” and section 5-302(c) to clarify that the posting is required “for the

duration of the time the AEDT is used” and for 6 months after that. Also, in the Bias Audit section, we

recommend changing Example #1 to mirror the existing language in Example #2 and clearly state that the audit is

required “within a year from the planned use of the AEDT.”

Publication of Audit Results: While well-intended, we believe a requirement to publish the specific selection rate

and impact ratio information may in practice undermine the goals of the regulation given the existing legal

landscape. Specifically, depending on the type of audit and data reviewed, it may require publication of

information that could appear to constitute adverse impact under existing employment law guidance. Notably,

existing anti-discrimination law acknowledges that the existence of adverse impact based on numerical

calculations alone is not unlawful under all circumstances. Thus, a summary statement requirement better aligns

with the letter and spirit of existing employment laws prohibiting discrimination, while also accomplishing the

goals of this ordinance.

To better achieve the desired outcomes of the regulation, we recommend striking “the selection rates and impact

ratios for all categories” in Section 5-302(a)(1) and replacing it with “a statement on adverse impact.”

Alternatively, we ask the Commission to consider leveraging language from the recently released White House

framing of algorithmic discrimination protections, and replacing the final clause of 503-2(a)(1) with “which may

include confirmation that if negative impacts were observed on any category, the employer or employment

agency has retained the relevant materials and justification for any continued use of the AEDT.”

TIMING OF NOTICE REQUIREMENT

Notifying Candidates: The language of Law 144 and the proposed regulation contemplate notice to candidates and

employees 10 days prior to use of an AEDT. It is not clear whether this applies to every separate individual

review of candidate/employee materials by an AEDT or whether it applies to the fact that an AEDT is in use in

the process, though the option to effect notice via policy or career website suggests the latter. If interpreted as

requiring 10 days-notice to each individual candidate before an AEDT could be used with respect to that

candidate information, the consequence would be significant for employer hiring processes, slowing down the

ability to hire by potentially placing candidates or employees in a 10 day “holding pen” before a resume could

even be reviewed. This would have a negative impact on hiring employers but also would have a significant

impact on job seekers.

To aid in clarifying this issue, we would recommend modifying the language of Section 5-303 to clarify that the

notice is required 10 days prior to placing an AEDT into use for the job in question, rather than 10 days prior to

each and every independent review by the AEDT.

CONCLUSION

LinkedIn appreciates the opportunity to provide comments on the proposed regulations implementing Local Law

144 governing automated employment decision tools. We look forward to working with the Department to

implement these rules in a way that supports the employers and jobseekers of NYC.

Sincerely,

Henry Fong

VP, Legal

LinkedIn Corporation

Enclosure

Enclosure accompanying: LinkedIn Comments on the New York City Department of

Consumer and Worker Protection’s Proposed Rules Implementing Local Law 144 of 2021 Regarding

Automated Employment Decision Tools

Proposed Rule Amendments

Section 1. Chapter 5 of Title 6 of the Rules of the City of New York is amended to add Subchapter T as

follows:

Subchapter T: Automated Employment Decision Tools §

5-300. Definitions.

As used in this subchapter, the following terms have the following meanings:

Automated Employment Decision Tool. “Automated employment decision tool” or “AEDT” means

“Automated employment decision tool” as defined by § 20-870 of the Code where the phrase “to

substantially assist or replace discretionary decision making” means to rely solely on a simplified output

(score, tag, classification, ranking, etc.), with no other factors considered, or to use a simplified output as

one of a set of criteria where the output is weighted more than any other criterion in the set, or to use a

simplified output to overrule or modify conclusions derived from other factors including human

decision- making.

Bias Audit. “Bias audit” means “Bias audit” as defined by § 20-870 of the Code.

Candidate for Employment. “Candidate for employment” means a person who has applied for a

specific employment position by submitting the necessary information and/or items in the format

required by the employer or employment agency.

Category: “Category” means any component 1 category required to be reported by employers pursuant

to subsection (c) of section 2000e-8 of title 42 of the United States code as specified in part 1602.7 of

title 29 of the code of federal regulations, as designated on the Equal Employment Opportunity

Commission Employer Information Report EEO-1.

Distribution Date. “Distribution date” means the date the employer or employment agency began using

a specific AEDT.

Employment Decision. “Employment decision” means “Employment decision” as defined by § 20-870

of the Code.

Employment Agency. “Employment agency” means “Employment agency” as defined by 6 RCNY § 5-

249.

Independent Auditor. “Independent auditor” means a person or group that is not involved in using or

developing an AEDT that is responsible for conducting a bias audit of such AEDT.

Impact Ratio. “Impact ratio” means either (1) the selection rate for a category divided by the selection

rate of the most selected category or (2) the average score of all individuals in a category divided by the

average score of individuals in the highest scoring category.

selection rate for a category

Impact Ratio = selection rate of the most selected category

average score of individuals in a category

Impact Ratio = average score of individuals in the highest scoring category

Machine learning, statistical modelling, data analytics, or artificial intelligence. “Machine learning,

statistical modelling, data analytics, or artificial intelligence” means a group of mathematical, computer-

based techniques:

i. that generate a prediction, meaning an expected outcome for an observation, such as an assessment of

a candidate’s fit or likelihood of success, or that generate a classification, meaning

an assignment of an observation to a group, such as categorizations based on skill sets or

aptitude; and ii. for which a computer at least in part identifies the inputs, the relative importance placed

on those inputs, and other parameters for the models in order to improve the accuracy of the prediction

or classification; and iii. for which the inputs and parameters are refined through cross-validation or by

using training and testing data.

Screen. “Screen” means to make a determination about whether someone a candidate or employee

should be selected or advanced in the hiring or promotion process.

Selection Rate. “Selection rate” means the rate at which individuals in a category are either selected to

move forward in the hiring process or assigned a classification by an AEDT. Such rate may be calculated

by dividing the number of individuals in the category moving forward or assigned a classification by the

total number of individuals in the category who applied for a position or were considered for promotion.

Example. If 100 Hispanic women apply for a position and 40 are selected for an interview after use of an

AEDT, the selection rate for Hispanic women is 40/100 or 40%.

Simplified output. “Simplified output” means a prediction or classification as specified in the definition

for “machine learning, statistical modelling, data analytics, or artificial intelligence”. A simplified output

may take the form of a score (e.g., rating a candidate’s estimated technical skills), tag or categorization

(e.g., categorizing a candidate’s resume based on key words, assigning a skill or trait to a candidate),

recommendation (e.g., whether a candidate should be given an interview), or ranking (e.g., arranging a

list of candidates based on how well their cover letters match the job description). It does not refer to the

output from analytical tools that translate or transcribe existing text, e.g., convert a resume from a PDF

or transcribe a video or audio interview.

§ 5-301 Bias Audit

(a) Where an AEDT selects individualscandidates or employees to move forward in the hiring process or

classifies those individuals into groups [in a way that prefers one group to move forward in a hiring or

promotion process], a bias audit required by § 20-871 of the Code must, at a minimum:

(1) Calculate the selection rate for each category;

(2) Calculate the impact ratio for each category; and

(3) Where an AEDT classifies individuals into groups as described above, the calculations in

paragraphs (1) and (2) of this subdivision must be performed for each such classification.

Example: An employer wants to use an AEDT to screen resumes and schedule interviews for a job

posting. To do so, the employer must ensure that a bias audit of the AEDT was conducted within a year

from the planned use of the AEDT. The employer asks the vendor for a bias audit. The vendor uses

historical data it has collected from employers on applicants selected for each category to conduct a bias

audit as follows:

# of

Applicants

Selected

Selection

Rate

Impact

Ratio

Hispanic or Latino

Male 205 90 43.9% 0.841

Female 190 82 43.2% 0.827

Non/Hispanic or

Latino

Male

White 412 52.2% 1.000

Black or African

American

226

42.0% 0.806

Native Hawaiian or Pacific

Islander

87 37 42.5% 0.815

Asian 321 167 52.0% 0.997

Native American or Alaska

Native

24 11 45.8% 0.878

Two or More Races 115

45.2% 0.866

Female

White 385 51.2% 0.981

Black or African

American

164 75 45.7% 0.876

Native Hawaiian or Pacific

Islander

32 15 46.9% 0.898

Asian 295 135 45.8% 0.877

Native American or Alaska

Native

17 7 41.2% 0.789

Two or More Races 98 44 44.9% 0.860

(b) Where an AEDT scores applicants or candidates, a bias audit required by § 20-871 of the Code must, at a

minimum:

(1) Calculate the average score for individuals in each category;

(2) Calculate the impact ratio for each category;

Example: An employer uses an AEDT to score applicants for “culture fit”. To do so, the employer must

ensure that a bias audit of the AEDT was conducted within a year from the planned use of the AEDT.

The employer uses historical data on average “culture fit” score of applicants for each category to

conduct a bias audit as follows:

Average

Score

Impact

Ratio

Hispanic or Latino

Male 76 0.884

Female 47 0.547

Non/Hispanic or

Latino

Male

White 86 1.000

Black or African American

76 0.884

Native Hawaiian or Pacific

Islander

65 0.756

Asian 71 0.826

Native American or Alaska

Native

69 0.802

Two or More Races 84 0.977

Female

White 68 0.791

Black or African American

56 0.651

Native Hawaiian or Pacific

Islander

74 0.860

Asian 67 0.779

Native American or Alaska

Native

52 0.605

Two or More Races 86 1.000

§ 5-302 Published Results

(a) Prior to theimplementing use of an AEDT, employers and employment agencies in the city must make the

following publicly available on the careers or jobs section of their website in a clear and conspicuous

manner:

(1) the date of the most recent bias audit of such AEDT and a summary of the results

, [including

statement on any adverse impact] OR [which may include confirmation that if negative

impacts were observed on any category the employer or employment agency has retained the

relevant materials and justification for any continued use of the AEDT] which shall include

the selection

rates and impact ratios for all categories

; and,

(2) the distribution date of the AEDT to which such bias audit applies.

(b) The requirements of subdivision (a) of this section may be met with an active hyperlink to a website

containing the required summary of results and distribution date, provided that such link is clearly

identified as a link to results of the bias audit.

the duration of the time that they use the AEDT and for at least 6 months after last using such AEDT for

an employment decision.

§ 5-303 Notice to Candidates and Employees

(a) For the purpose of compliance with § 20-871(b)(1) and (2) of the Code, an employer or employment

agency may provide notice to a candidate for employment who resides in the city by:

(1) Including notice on the careers or jobs section of its website in a clear and conspicuous manner

at least 10 business days prior to use of anplacing an AEDT into use;

(2) Including notice in a job posting at least 10 business days prior to placing use of an AEDT into

use for the relevant job; or,

(3) Providing notice to candidates for employment via U.S. mail or e-mail at least 10 business days

prior to use of an placing an AEDT into use for the relevant job.

(b) For the purpose of compliance with § 20-871(b)(1) and (2) of the Code, an employer or employment

agency may provide notice to an employee who resides in the city by:

(1) Including notice in a written policy or procedure that is provided to employees at least 10

business days prior to use of placing an AEDT into use;

(2) Including notice in a job posting at least 10 business days prior to use of an AEDT for the

relevant job; or,

(3) Providing written notice in person, via U.S. mail or e-mail at least 10 business days prior to

use of an AEDT.

alternative selection process or accommodation.

(d) For the purpose of compliance with § 20-871(b)(3) of the Code, an employer or employment agency

must:

(1) provide notice to an employee or candidate who resides in the city by:

i. Including notice on the careers or jobs section of its website in a clear and conspicuous

manner; or,

ii. Providing written notice in person, via U.S. mail or e-mail within 30 days of receipt of

a written request for such information.

(2) where notice is not included on its website, post instructions for how to make a written request

for such information on the careers or jobs section of the employer or employment agency’s

website in a clear and conspicuous manner.

(3) Provide an explanation to a candidate or employee for refusing to provide information pursuant

to § 20-871(b)(3) of the Code where disclosure of information required by the notice would

violate local, state, or federal law, or interfere with a law enforcement investigation.

(e) Nothing in this subchapter requires an employer or employment agency to provide an alternative

selection process

Consumer Data Industry Association 1090 Vermont Ave., NW,

Suite 200 Washington, D.C. 20005-4905

P 202 371 0910 Writer’s direct dial: +1 (202) 408-7407

October 24, 2022

CDIAONLINE.ORG

The Honorable Vilda Vera Mayuga

Commissioner, New York City Department of Consumer and Worker Protection Consumer Services

Division

42 Broadway, 9th Floor

New York, NY 10004

Via Email: Rule[email protected]

Re: Proposed rules relating to the use of automated employment decision tools

Dear Commissioner Maygua:

I write on behalf of the Consumer Data Industry Association (CDIA) to offer our comments to the Department of

Consumer and Worker Protection (“DCWP” or “Department”) on proposed rules relating to the use of automated

employment decision tools. The proposed rules are being issued under Local Law 144 (2021) (“LL 144”).

An employer’s use of augmented intelligence is rarely a binary decision. When used appropriately by employers,

a combination of human and technology tools can help reduce bias and get people to work more quickly in ways

that are fair, legal, and replete with second chances. Technology in hiring can reduce unintentional discrimination

The Consumer Data Industry Association (“CDIA”) is the voice of the consumer reporting industry, representing consumer reporting agencies

including the nationwide credit bureaus, regional and specialized credit bureaus, background check and residential screening companies, and others.

Founded in 1906, CDIA promotes the responsible use of consumer data to help consumers achieve their financial goals, and to help businesses,

governments and volunteer organizations avoid fraud and manage risk. Through data and analytics, CDIA members empower economic opportunity all

over the world, helping ensure fair and safe transactions for consumers, facilitating competition and expanding consumers’ access to financial and other

products suited to their unique needs

in employment and reduce the chances that human error can create a barrier to employment. Technology in hiring

meets the urgent need to hire quickly, hire smartly, and hire remotely. Any rule from DCWP must be carefully

balanced to meet the needs of consumers as asserted in the passage of LL 144, with the needs of employers and

employees to higher quickly. The proposed rule has the potential to slow down the hiring process across the City.

People should be able to have confidence that automated decision tool are lawfully compliant, and are used

responsibly in their development, deployment, and use. Given the speed of technology, trust in automated

decision tools is a shared responsibility of government and the private sector. Governance frameworks should be

flexible and driven by a voluntary, and multistakeholder process. With those considerations in mind, CDIA offers

comments on the proposed rule.

Definition of an automated decision tool (§ 5-300)

CDIA respectfully requests three changes to this definition. First, we request that the Department strike “or to use

a simplified output to overrule or modify conclusions derived from other factors including human decision

making” and substitute in its place the following “’Automated employment decision tool,’ or ‘AEDT,’ does not

include (a) the automated searching of resumes to identify candidate qualifications, including relevant skills or

experience, or (b) tools that merely covert a manual process to an automated process without the introduction of

machine learning or artificial intelligence, and does not include tools that do not result in a final employment

decision.”

Most employers do not use AEDT as the only factor in determining who to employ. Employers take a more

holistic approach. For example, just because an employer may use an algorithm on every potential candidate, it

does not mean that that tool is always used. While an AEDT may review each candidate, the discretion may still

rest with a hiring manager to determine whom to interview, as well as the potential reliance and weight placed on

the use of the tool.

Bias audit (§ 5-301)

We respectfully request that the examples offered in the proposed rule clarified so that, among other things, they

should not necessarily represent an exhaustive list of all the possible scenarios that might be available to

employers. We ask the Department to remove the specificity of responsibility in each of the two examples of a

bias audit to bring flexibility to account for a myriad of employment scenarios.

The examples provided in Subsections (a) and (b) are prescriptive in noting who bears responsibility for the bias

audit (i.e., the employer/deployer or the vendor/developer). Yet, the examples fair to cover a range of possible

hiring scenarios. For this reason, the examples fall short of guidance for employers.

We request a change to section (a): “Where an AEDT is used to Screen a candidate or employee for an

employment decision, a bias audit required by § 20-871 of the Code must, at a minimum:” We also request that

the Department change Subsection (a) in the example, strike “historical data” and replace it with “test data.” In

Subsection (b) we ask that the Department strike “historical data” and replace it with “test data.” In the model we

request that the Department strike “planned.”

Published results (§ 5-302)

We respectfully ask for two changes. First, an addition of the following in section (a) “Prior to the use of an

AEDT to screen a candidate or employee for an employment decision, employers and employment agencies in

the city must make the following publicly available on the careers or jobs section of their website in a clear and

conspicuous manner:” Second, we request that the Department strike “the selection rates and impact ratios for all

categories” and replace with “a statement on adverse impact.”

Definition of screen in relation to employment decisions (§ 5-300)

Local Law 144 defines an “employment decision” as a “screen [of] candidates for employment or employees for

promotion within the city.” § 20-870. We request clarity on how the use of some A.I. tools would fit within this

definition. For example, we do not believe that if a tool makes a recommendation about an individual, but that

recommendation is subject to human review, that that recommendation qualifies as a “determination.” Further,

since LL 144 does not extend beyond the limits of New York City, we encourage the Department to clarify the

proposed rule to state that “within the city” means positions located in the City of New York, regardless of where

the employee resides.

Independent auditor (§ 5-301)

The proposed rule defines an independent auditor as “a person or group that is not involved in using or

developing an AEDT.” We request that the proposed rule confirm the implication that an employer or vendor

can conduct a bias audit with internal employees, provided that the group conducting the audit is not involved in

using or developing the specific tool.

Obligation to provide an alternative selection method (§ 5-303)

Local Law 144 says that required notices to candidates must allow them “to request an alternative selection

process or accommodation.” § 20-871.b.1. But, the proposed rule states that nothing in the rules “requires an

employer or employment agency to provide an alternative selection process.” We read that as saying employers

must provide an opportunity to request an alternative selection process in their notices. This requirement is a

challenge because employers are never obligated to provide the alternative. The rule should confirm this logical

interpretation.

Frequency of audits

Local Law 144 says that an AEDT cannot be used unless a bias audit was “conducted no more than one year

prior to the use of such tool.” § 20-871.b.1. It is not clear whether that language requires yearly bias audits of the

tool or if conducting one audit on a tool is sufficient unless or until the tool is replaced or materially modified.

The Statement of Basis and Purpose of the Proposed Rule states that a bias audit is required “within one year of

use of the tool.” This clause implies that the audit may take place within the 12 months which follow

implementation of the tool. Since this requirement was not the intent of the proposal, it should be clarified.

Grace period

While we understand the legislation requires the rules and regulations to go into effect by January 2023. We

strongly encourage the Department to provide a grace period to businesses and organizations as they look to

implement the final rule.

Summary

CDIA appreciates the opportunity to comment on the implementing rules. An employer’s use of augmented

intelligence is rarely a binary decision. When used appropriately by employers, a combination of human and

technology tools can help reduce bias and get people to work more quickly in ways that are fair, legal, and replete

with second chances. Any rule from DCWP must be carefully balanced to meet the needs of consumers as

asserted in the passage of LL 144, with the needs of employers and employees to higher quickly. The proposed

rule has the potential to slow down the hiring process across the City.

Sincerely,

Eric J. Ellman

Senior Vice President, Public Policy & Legal Affairs

TechNet Northeast | Telephone 774.230.6685

One Beacon Street, Suite 16300, Boston, MA 02108

www.technet.org | @TechNetNE

October 24, 2022

New York City Department of Consumer and Worker Protection

Re: Proposed Rules Relative to the Use of Automated Employment Decision Tools

Thank you for the opportunity to submit comments regarding DCWP’s proposed rules

implementing Local Law 144 of 2021, regarding automated employment decision tools

(AEDT).

TechNet is the national, bipartisan network of technology CEOs and senior executives that

promotes the growth of the innovation economy by advocating a targeted policy agenda at

the federal and 50-state level. TechNet's diverse membership includes dynamic American

businesses ranging from startups to the most iconic companies on the planet and represents

over five million employees and countless customers in the fields of information technology,

e-commerce, the sharing and gig economies, advanced energy, cybersecurity, venture

capital, and finance. TechNet has offices in Austin, Boston, Chicago, Denver, Harrisburg,

Olympia, Sacramento, Silicon Valley, and Washington, D.C.

TechNet’s membership includes both employers in NYC and companies that make some

of the tools that may be regulated under these rules. We are interested in ensuring clarity

in the roles and responsibilities of each party, and whether the test of applicability is the

location of the posted position or posting company, or the location in which the tool was

used. We further hope to focus the proposed rules on the kinds of tools and use cases that

may actually have a meaningful, measurable impact on final employment-related

decisions. Additionally, we seek clarity on the interoperability of the proposed rules with

existing state and federal laws and standards, including the Equal Employment

Opportunity Act.

Definition of AEDT

TechNet remains concerned that the definition of AEDT in LL144 is drafted so broadly as

to encompass simple organizational tools like keyword search.

Particularly when a posting attracts a large pool of applicants, these basic tools can be

crucial in scanning cover letters and resumes for requisite core competencies listed in the

job description.

To help address that concern, we ask that the Department consider further tightening the

proposed definition of AEDT as follows:

Austin • Boston • Chicago • Denver • Harrisburg • Olympia • Sacramento • Silicon Valley • Washington, D.C.

Automated Employment Decision Tool. “Automated employment decision tool”

or “AEDT” means “Automated employment decision tool” as defined by § 20-870

of the Code where the phrase “to substantially assist or replace discretionary

decision making” means to rely solely on a simplified output (score, tag,

classification, ranking, etc.), with no other factors considered, or to use a simplified

output as one of a set of criteria where the output is weighted substantially more

than any other criterion in the set., or to use a simplified output to overrule or modify

conclusions derived from other factors including human decision making.

“Automated employment decision tool” or “AEDT” does not include (1) tools to

help improve the overall quality of candidates for employment or the efficiency of

the recruiting process, prior to a natural person applying for a specific employment

position; (2) the automated searching of resumes or other relevant documents to

identify candidate qualifications including relevant skills or experience; or (3)

filtering or automated labeling of candidates to help identify qualifications

Definition of Screening

Ultimately, the proposed definition of AEDT in the proposed rule hinges more on the

human use of the tool than the essence of the tool itself. In order to align the rest of the

rules with the intent signaled in that definition, we propose an adjustment to the definition

of “screen” to focus the rules on the processes that may have concrete effects on decision

making:

Screen. “Screen” means to make a determination that a candidate for employment

should not be hired or that an employee should not be granted a promotion. about

whether someone should be selected or advanced in the hiring or promotion process.

We further recommend clarifications throughout the rules to specify that the bias audit

requirement is triggered when an employer uses an AEDT for screening purposes.

Bias Audit

When it is determined that an AEDT has been used for screening purposes, it is unclear to

TechNet exactly which party – the prospective employer or the maker of the AEDT – is

responsible for performing the required bias audit. We are concerned that both parties

may experience significant operational difficulties in complying with the rules as

currently proposed.

Under existing federal standards, the demographic information required in the proposed

bias audit may be unavailable or uninstructive depending on which party is deemed

responsible. Under the Equal Employment Opportunity Act, the provider of the AEDT is

not required to collect demographic information, while most (but not all) prospective

employers are.

Furthermore, employers cannot require applicants to provide that information, generally

offering an option like “prefer not to say”. In this case, in order to comply with the

proposed NYC rule, is the Department proposing that relevant parties will infer the

missing information or compel applicants to provide information that is not currently

required under existing standards?

If certain groups opt out of providing demographic information at higher rates than

others, the resulting audits would be inaccurate and misleading. This problem would be

particularly acute in smaller sample sizes, and is compounded by the requirements to

conduct intersectional analysis, which further diverges from existing US standards. If

companies have a small pool of relevant job postings or applicants, aberrations that are

not otherwise statistically significant could paint a picture of an employer that does not

accurately reflect reality. Would the Department consider a trigger that ensures these

audits would be required only in cases where the volume or quality of data available

would produce useful and actionable audits that provide statistically significant

information?

Enforcement

In the interest of addressing the many open questions, the potential difficulties of

operationalizing this new rule, and the length of time that it may take to conduct a bias

audit compliant with the proposed rules, we ask that the Department consider a delay in

enforcement for at least 6 months after final rules are promulgated.

Thank you for your consideration. TechNet’s members are eager to be of service as the

Department considers these complex issues. Please do not hesitate to contact me if I can

provide any additional information.

Sincerely,

Christopher Gilrein

Executive Director, Northeast

TechNet

[email protected]

Legislative Affairs

One Whitehall Street

New York, NY 10004

212-607-3300 www.nyclu.org

Testimony of Daniel Schwarz

On Behalf of the New York Civil Liberties Union

Before the New York City Department of Consumer and Worker Protection Regarding the Proposed Rules

to Implement Local Law 144 of 2021

October 24, 2022

The New York Civil Liberties Union (“NYCLU”) respectfully submits the following testimony regarding the

proposed rules to implement Local Law 144 of 2021. The NYCLU, the New York affiliate of the American Civil

Liberties Union, is a not-for-profit, non-partisan organization with eight offices throughout the state and more

than 180,000 members and supporters. The NYCLU’s mission is to defend and promote the fundamental

principles, rights, and values embodied in the Bill of Rights, the U.S. Constitution, and the Constitution of the

State of New York. The NYCLU works to expand the right to privacy, increase the control individuals have over

their personal information, and ensure civil liberties are enhanced rather than compromised by technological

innovation.

The New York City Council enacted Local Law 144 of 2021 (“LL 144”) which laudably attempts to tackle bias in

automated employment decision tools (“AEDT”). AEDT, similar to automated decision systems in other areas,

are in urgent need of transparency, oversight, and regulation.

These technologies all too often replicate and

amplify bias, discrimination, and harm towards populations who have been and continue to be disproportionately

impacted by bias and discrimination: women, Black, Indigenous, and all people of color, religious and ethnic

minorities, LGBTQIA people, people living in poverty, people with disabilities, people who are or have been

incarcerated, and other marginalized communities. And the use of AEDT is often accompanied by an acute power

imbalance between those deploying these systems and those affected by them, particularly given that AEDT

operate without transparency or even the most basic legal protections.

Unfortunately, LL 144 falls far short of providing comprehensive protections for candidates and workers. Worse,

the rules proposed by the New York City Department of Consumer and Worker Protection (“DCWP” or

“Department”) would stymie the law’s mandate and intent further by limiting its scope and effect.

The DCWP must strengthen the proposed rules to ensure broad coverage of AEDT, expand the bias audit

requirements, and provide transparency and meaningful notice to affected people in order to ensure that AEDT do

See: Testimony on Oversight and Regulation of Automated Decision Systems, NEW YORK CIVIL LIBERTIES UNION (2020),

https://www.nyclu.org/en/publications/testimony-oversight-and-regulationautomated-decision-systems.

not operate to digitally circumvent New York City’s laws against discrimination. Candidates and workers should

not need to worry about being screened by a discriminatory algorithm.

Re: § 5-300. Definitions

LL 144 defines AEDT as tools that “substantially assist” in decision making. The proposed rules by DCWP

further narrow this definition to “one of a set of criteria where the output is weighted more than any other criterion

in the set.” This definition goes beyond the law’s intent and meaning, risking coverage only over certain scenarios

and a subset of AEDT. In the most absurd case, an employer could deploy two different AEDT, weighted equally,

and neither would be subject to this regulation. More problematically, an employer could employ an AEDT in a

substantial way that doesn’t meet this threshold, while still having significant impact on the candidates or

workers.

The Department should revise this definition to be consistent with the statute.

The proposed definition of “simplified output” would exclude “output from analytical tools that translate or

transcribe existing text, e.g., convert a resume from a PDF or transcribe a video or audio interview.” However,

existing transcription tools are known to have racial bias,

and their outputs could very well be used as inputs to

other AEDT systems, resulting in biased results.

Re: § 5-301 Bias Audit

The definition for bias the audit in LL 144, § 20-870, explicitly lists disparate impact calculation as a component

but not the sole component (“include but not be limited to”). follows. The examples given in section 5-301 of the

proposed rules do not account for an AEDT’s impact on age and disability, or other forms of discrimination.

At a minimum, in addition to an evaluation of disparate impact of the AEDT, any evaluation that could properly

qualify as a bias audit would need to include an assessment of:

• the risks of discriminatory outcomes that an employer should be aware of and control for with the

specific AEDT, including risks that may arise in the implementation and use of the AEDT;

• the sources of any training/modeling data, and the steps taken to ensure that the training data and samples

are accurate and representative in light of the position’s candidate pool;

• the attributes on which the AEDT relies and whether it engages in disparate treatment by relying on any

protected attribute or any proxy for a protected attribute;

• what less discriminatory alternative inputs where considered and which were adopted;

• the essential functions for each position for which the AEDT will be used to evaluate candidates, whether

the traits or characteristics that the AEDT measures are necessary for the essential functions, and whether

the methods used by the AEDT are a scientifically valid means of measuring people’s ability to perform

essential job functions.

Similar essential components are outlined in the federal EEOC guidance, which recommends including

“information about which traits or characteristics the tool is designed to measure, the methods by which those

Aaron Rieke & Miranda Bogen, Help Wanted, UPTURN (2018), https://upturn.org/work/help-wanted/.

Allison Koenecke et al., Racial disparities

in automated speech recognition, 117 PNAS 7684 (2020).

See: Algorithmic Discrimination Protections, THE WHITE HOUSE OSTP (2022), https://www.whitehouse.gov/ostp/ai-bill-of-rights/algorithmic-

discrimination-protections-2/.

traits or characteristics are to be measured, and the disabilities, if any, that might potentially lower the assessment

results or cause screen out.”

The bias audit should clearly state the origin of the data used for the statistics reported. This includes where the

data was gathered from, by who, when, and how it was processed. It should also provide justification for why the

source of the data for the bias audit model population is believed to be relevant to this specific deployment of the

AEDT.

The proposed rules for the ratio calculations also make no mention of appropriate cutoffs when a specific

candidate category (per EEO-1 Component 1) has a small or absent membership that could result in

unrepresentative statistics.

LL 144 mandates that any “tool has been the subject of a bias audit conducted no more than one year prior to the

use of such tool.” The DCWP’s rules should clarify that this requires annual auditing for continued AEDT use and

provide an example similar to the rate calculation scenarios.

Re: § 5-302 Published Results

The disclosure of the bias audit on employers’ and employment agencies’ websites should not be limited to the

selection rates and impact rates results described in §5-301. It should include all the elements mentioned in our

comments on the bias audit. The summary should describe the AEDT appropriately and include information on

traits the tool is intended to assess, the methods used for this, the source and types of data collected on the

candidate or employee, and any other variables and factors that impact the output of the AEDT. It should state

whether any disabilities may impact the output of the AEDT.

Additionally, the published results should list the vendor of the AEDT, the specific version(s) of product(s) used,

and the independent auditor that conducted the bias audit. The DCWP should provide examples that include such

information.

The “distribution date” indicated in the proposed rules for the published results should also describe which

particular part of the employment or promotion process the AEDT is used for on this date. It is insufficient to note

“an AEDT with the bias audit described above will be deployed on 2023-05-21” unless there are already clear,

public indicators that describe which specific employment or promotion decision-making process happened on

that date. Any examples should be updated to include a reasonable deployment/distribution description.

Published results should include clear indicators about the parameters of the AEDT as audited and testing

conditions, and the regulations should clarify that employers may not use the AEDT in a manner that materially

differs from the manner in which the bias audit was conducted. This includes how input data is gathered from

candidates or employees compared to how the comparable input data was gathered from the model population

used for the bias audit. For example, if testing of an AEDT used a specific cutoff or weighting scheme, the cutoff

or weighting scheme used in the actual deployment should match it as closely as possible, and the publication

should indicate any divergence and the reason for it. A tool may not show a disparate impact when cut offs or

rankings are set at one level and show a disparate impact with other levels. Likewise, if one input variable is hours

worked per week, the model population for the bias audit derives those figures from internal payroll data, but

candidate data will come from self-reporting, then the publication should indicate that divergence and provide

commentary on the reason for the divergence and an assessment of the impact the divergence is likely to have on

the relevance of the bias audit.

The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to

Assess Job Applicants and Employees, US EEOC (2022), https://www.eeoc.gov/laws/guidance/americans-disabilities-act-and-use-software-algorithms-

andartificial-intelligence.

Lastly, the rules should clarify that the published results must be disclosed in machine readable and ADA

compliant formats in order to be accessible to people with various assistive technologies.

Re: § 5-303 Notice to Candidates and Employees

Section 20-871(b)(2) of LL 144 requires the disclosure of the job qualifications and characteristics that the AEDT

will use in the assessment. The rules should clarify that candidates or employees should be provided with as much

information as possible to meaningfully assess the impact the AEDT has on them and whether they need to

request an alternative selection process or accommodation.

The law also requires that the employer “allow a candidate to request an alternative selection process or

accommodation.” The regulations should provide employers with parameters of how to provide alternative

selection processes or accommodations, including what processes may be used to give equal and timely

consideration to candidates that are assessed with accommodations or through alternative processes. By merely

stating in the regulations that “Nothing in this subchapter requires an employer or employment agency to provide

an alternative selection process,” the regulations suggest that the law provides an empty protection to candidates

to be solely allowed to make a request for an alternative without any obligation on the part of the employer to in

any way consider or honor that request.

Conclusion

In conclusion, the NYCLU thanks the Department of Consumer and Worker Protection for the opportunity to

provide testimony. The Department’s rulemaking is instrumental in ensuring a productive implementation of

Local Law 144 and making clear that discriminatory technology has no place in New York. We strongly urge the

Department to amend and strengthen the proposed rules to deliver on the law’s promise to mitigate bias and

provide people with the information they need.

See: Algorithms, Artificial Intelligence, and Disability Discrimination in Hiring, BETA.ADA.GOV (2022), https://beta.ada.gov/resources/ai-

guidance/.

Comments of the Partnership for New York City to the Department of Consumer and

Worker Protection

Proposed Rules Implementing Local Law 144 of 2021

October 24, 2022

Thank you for the opportunity to comment on the proposed rules implementing Local Law 144 of

2021 (LL144) regulating automated employment decision tools (AEDT). The Partnership for New

York City represents private sector employers of more than one million New Yorkers. We work

together with government, labor, and the nonprofit sector to maintain the city’s position as the

preeminent global center of commerce, innovation, and economic opportunity.

The Partnership appreciates the work the Department of Consumer and Worker Protection has

done to craft these proposed rules. Below are recommended changes and questions:

§ 5-300 – Definition of Automated Employment Decision Tool

• Issue: Applying the law to tools that simply “modify” conclusions goes beyond what is

required by LL144. It also leads to substantial confusion because it is unclear whether the use of

a tool’s results as one factor in a human’s decision would be considered to

“modify conclusions derived from other factors including human decision-making.” o For

example, does a screening tool count as an AEDT when used as follows? The tool compares an

applicant’s resume and application to the posted job description and provides a score reflecting

how closely the applicant matches the job description. The score is used to prioritize applicants

for review by a recruiter, however the recruiter reviews the resumes and applications for all

applicants. The recruiter determines whether the applicant is qualified to move forward in the

hiring process.

o Recommendation: Strike “or modify”.

§ 5-300 – Definition of Independent Auditor

• It is not clear whether an independent auditor must be external to the employer.

o Recommendation: Clarify that a group within a company that does not use the AEDT may

serve as an independent auditor for the AEDT used by another group in the company. This is

especially important for companies that develop and use AEDTs.

§ 5-301 – Bias Audit

• Employers who use any kind of tests in employment are already required by the federal

Equal Employment Opportunity Commission (EEOC) to conduct an adverse (a.k.a.

disparate) impact analysis on the test. The analysis requires employers to make comparisons

between a minority group and a majority group (e.g., women vs. men) but does not require the

analysis to include intersectional subgroups (e.g., Black men vs. Caucasian men). Including

intersectional groups requires substantially more work and may result in meaningless output

where specific population sizes are too small.

o Recommendation: The rules should mirror the EEOC’s adverse impact analysis requirements.

• The proposed rule appears to require employers or vendors to use actual or historical data to

conduct the bias audit. This has two problems:

o Employers may request job applicants disclose demographic data voluntarily. If applicants do

not volunteer this information, the bias audit results may be inaccurate. This would be

particularly true if certain categories of applicants refuse to provide the information at higher

rates.

o Requiring the use of historical data (as in the example in §5-301(a)) implies that the AEDT has

previously been used. This would preclude the use of new tools.

o Recommendations: Allow the bias audit to be conducted with “test data” instead of actual or

historical data.

 If the final rules do not allow the use of test data, they should clarify whether the data

used in the bias audit should be from all applicants or New York City residents only.

• It is unclear whether an employer must conduct the §5-301(a) analysis or the §5-301(b) analysis

if the employer uses an AEDT to create a score that selects individuals to move forward in the

hiring process.

o Recommendation: Clarify which test the employer is required to conduct.

• The examples provided in §§5-301(a) and (b) are specific about who performs the bias audit and

do not account for all of the possible actors who could perform the audit in each example.

Although these are listed as examples, this may still be confusing to employers.

o Recommendation: Remove the reference to the actor performing the audit (“the vendor” in (a)

and “the employer” in (b)). It would also be helpful to include a statement that the examples do

not show all of the possible options for the audit process.

• The example in §5-301(a) suggests that the employer can ask the vendor providing the AEDT to

conduct the bias audit. Since it is common for the AEDT vendor to also be the developer,

allowing the vendor to conduct the bias audit appears to conflict with the definition of

“Independent auditor” (i.e., “a person or group not involved in using or developing the AEDT”).

o Recommendation: Clarify whether a vendor who is also the developer can be considered an

independent auditor.

• In the example in §5-301(b), the word “planned” is confusing. LL144 requires an AEDT to have

been audited within one year of use, not within one year of planned use.

o Recommendation: Delete the word “planned.”

§5-302 – Published Results

• Publishing the impact ratio and/or selection rate does not provide the applicant with a

way to assess the existence of bias. Moreover, LL144 only requires a summary of results, not the

publication of the calculations used to conduct a bias audit.

o Recommendation: Require a “statement on adverse impact” which is more akin to a summary

than specific rates.

§5-303 – Notice to Candidates and Employees

• Although §§5-303(a)(3) and (b)(3) allow an employer to provide individual notices only

to job applicants who live in New York City, this is not administratively feasible (especially if

the employer is required to wait 10 days to use the AEDT after providing notice).

o Questions:

 Is the intent to require employers to provide the notice to all applicants for all jobs, regardless of

where they are located?

 Is the employer obligated to wait 10 days between notifying the applicant and administration of

the AEDT?

 Could an applicant be provided a notification that includes a choice to take an assessment that is

an AEDT within the 10-day period?

Effective Date

• The final rules seem likely to be released shortly before the January 1, 2023 effective date

for LL144. This will not leave employers with sufficient time to implement the rules.

o Recommendation: The Department should provide an extended time for employers to

implement the rules before enforcing the law.

Thank you for considering our comments and questions.

From:

Davies, Rachael <Rachael.Davies@claconnect.com>

Sent:

Tuesday, October 11, 2022 5:26 PM

To:

Rulecomments

Subject:

[EXTERNAL] Clarifications on Rules set forth for upcoming AI legislation

Some people who received this message don't often get email from rachael.davies@claconnect.com. Learn why this is important

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize

the sender and know the content is safe. Forward suspect email to phish@oti.nyc.gov as an attachment (Click the More button,

then forward as attachment).

Hello,

To whom it may concern I have the following questions I hope may be addressed at the upcoming comment session on

Oct. 24

1) In regard to the independent bias audit

a. Does the audit need to be done on candidate data that is specific to that employer’s use of the tool, or

can the audit be done on the tool as a whole. Ie the AI vendor’s data for its use across multiple

employers?

b. If the bias audit needs to be run on candidate data from the specific employer, how would one acquire

said candidate data if the tool is not yet in use due to the requirement to run a bias audit ahead of its

implementation?

2) In regard to the 10 day notice requirement

a. Does that refer to 10 business days prior to the tool’s general use? Or is it 10 business days’ notice prior

to the use for each specific role? Meaning a requisition would need to remain open and posted at

minimum 10 business days prior to reviewing any of the candidate’s since they are entitled to 10 days

notice before the use of the tool?

Thanks in advance for the clarification

To help protect your privacy, Micr osoft Office pr evented auto mac Rachael Davies (she/her/hers)

download of this p icture

Employment Practices Senior People Solutions

Direct 626-844-2035

CLA (CliftonLarsonAllen LLP)

rachael.davies@CLAconnect.com

We'll get you there.

CPAs | Consultants | Wealth Advisors

Send me your files with secure file transfer.

CLA (CliftonLarsonAllen LLP) is an independent network member of CLA Global. See CLAglobal.com/disclaimer. Investment advisory services are

offered through CliftonLarsonAllenWealth Advisors, LLC, an SEC-registered investment advisor.

1 155 Connecttcut Ave NW

Suite 900

resolution economics Washington, D.C. 20036

(202) 618-4470

Resolution Economics, LLC

October 24, 2022

Hon. Vilda Vera Mayuga, Esq.

Commissioner

Department of Consumer and Worker Protection

42 Broadway, 9

Floor

New York, NY 10004

[email protected]c.gov (By Email Only)

RE: Comments to Proposed Rules for Local Law 144 of 2021

Dear Commissioner Mayuga:

Resolution Economics, LLC, an international consulting firm with offices in New York, Los

Angeles, Chicago, Washington, DC, and Charlotte, NC, makes this submission in response to

the Notice of Public Hearing and Opportunity to Comment that was issued by the New York

Department of Consumer and Worker Protection regarding the Proposed Rules implementing

the Automated Employment Decision Tools law ("AEDT law") that goes into effect on January

1, 2023.

Resolution Economics provides economic and statistical analysis, investigations and advisory

services, tailored technology, and analytical solutions as well as expert testimony to law firms,

companies, and government agencies. We specialize in global labor and employment and

financial services matters, including in litigation settings, across every industry. Our

professionals include highly trained and technical team members with PhDs, JDs, MAs, MBAs,

CPAs, CFEs, and other qualifying expertise. Resolution Economics has been and is currently

advising employers on how to evaluate the impact and navigate compliance obligations around

automated employment decision tools.

Based on our review of the AEDT law and proposed regulations and our experience in this area,

Resolution Economics submits the following questions regarding the Proposed Rules:

1. What type of analysis is required in order to be in compliance with the bias audit

requirement and not subject to penalties under the AEDT?

Section 5-301 of the Proposed Rules raises several issues regarding the type of analysis

required for a "bias audit."

a. That Section indicates that a bias audit must "calculate the impact ratio"

for certain categories. (5 5-301(a) and (b)). The definition of "bias audit" in Local Law

2021/144, however, specifies that the bias audit must include testing "to assess the tool's

disparate impact on persons" within those categories. (AEDT law 20-870).

The concept of "disparate impact" encompasses a much broader analysis than simple

impact ratio calculations. How are these two different types of analyses to be reconciled? More

specifically: If an impact ratio is outside the 0.8 and 1.25 range, is use of the tool automatically a

violation of the law that subjects the employer to penalties pursuant to 20-872 of the AEDT law?

Or does the employer have the opportunity to show that there are business or job related

legitimate reasons for the observed differences, as would occur under disparate impact analysis?

b. The impact ratio tables (S 5-301(a) or (b)) given as an example in the

Proposed Rules calculate the impact ratios for the intersection of race and gender

categories. Does this mean that race-only and gender-only impact ratios are not required?

2. What level of specificity is required for a bias audit?

Generally, an analysis to determine whether a selection tool has a particular impact must

be performed on a use case basis — that is, be based on data that is specific to a certain

population and time period and that is specific to the employer's particular application of the

tool. For example, if an employer were using a selection tool in relation to filling two positions

at different times — say, one for a front desk clerk in February and another for a warehouse

worker six months later separate analyses would need to be performed for each of these different

applications of the tool.

It is not clear if the Proposed Rules require this type of analysis for AEDTs or instead

permit some sort of single "generic" study that applies to all uses of a given tool. The example

given under 5-301 (a) says "The employer asks the vendor for a bias audit. The vendor uses

historical data it has collected from employers on applicants selected for each category to

conduct a bias audit...". Does this mean that the vendor may use historical data collected from

other employers that used this AEDT to carry out the bias audit? Furthermore, may different

employers that plan to use the AEDT from the same vendor use the same bias audit conducted

by the vendor using historical data from other employers to comply with the proposed rule? Or

is it required that the data used for a bias audit be specific to a particular employer and to the

specific application of the AEDT at issue?

The language of 5-302(a) (Published Results) raises this same issue. It states that

employers must make the results of an AEDT bias audit publicly available prior to use of the

AEDT. It would seem, however, that an employer would need to first implement an AEDT for

some period of time to collect the data necessary as an input to the bias audit. Is it the case,

therefore, that an employer needs to conduct some sort of pilot study with the AEDT prior to

relying upon it to make employment decisions? Or may the employer (a) rely on a generic audit

done by the vendor; and/or (b) use synthetic data — that is data that is generated partially or

completely artificially, rather than being measured or extracted from real-world events — to

conduct a bias audit prior to actual use of the AEDT?

3. What level of autonomy is required for the person/group performing the bias audit?

The AEDT Law states that the bias audit must be done by "an independent auditor." (5

20-870). Although the AEDT Law does not provide a definition of "independent," Merriam

Webster defines independent as "(1): not subject to control by others; self-governing; (2): not

affiliated with a larger controlling unit." In the context of financial statement audits by public

accounting firms, employees of an audited entity are not generally considered to qualify as

"independent auditors." See, e.g., 17 C.F.R. 210.2-01(c)(2) ("An accountant is not independent

if, at any point during the audit and professional engagement period, the accountant has an

employment relationship with an audit client").

The Proposed Rules define "independent auditor" as "a person or group that is not involved

in using or developing an AEDT that is responsible for conducting a bias audit of such AEDT."

Where an employer is required to conduct a bias audit, does the Department intend this definition

to allow the bias audit to be performed by an employee or group of employees of the employer?

Thank you for considering these questions.

Paul White, PhD

Partner

Resolution Economics, LLC

Washington, DC

Cc: Kevin Bandoian, CPA, Resolution Economics, New York, NY

Tricia Etzold, CPA, Resolution Economics, New York, NY

Rick Holt, PhD, Resolution Economics, Washington, DC

Victoria A. Lipnic, JD, Resolution Economics, Washington, DC

Ali Saad, PhD, Resolution Economics, Los Angeles, CA

October 24, 2022

Commissioner Vilda Vera Mayuga

New York City Department of Consumer and Worker Protection

42 Broadway, 8th Floor New

York, New York 10004

http://rules.cityofnewyork.us

[email protected]c.gov

Re: Comments Regarding Proposed Rules to the Use of Automated Employment

Decision Tools Under Local Law 144 of 2021

Dear Commissioner Mayuga:

The Society for Human Resources Management (“SHRM”) is the nation’s preeminent HR trade

association. More than 95% of Fortune 500 companies rely on SHRM as their go-to resource for

all things work and their business partner in creating next-generation workplaces. With more

than 300,000 HR and business executive members in 165 countries, SHRM enhances the lives of

more than 115 million workers and families globally. SHRM creates better workplaces where

businesses and workers thrive together. As the voice of all things work, workers, and the

workplace, SHRM is the foremost expert, convener, and thought leader on issues impacting

today’s evolving workplaces. The SHRM vision is to build a world of work that works for all.

Employers, like any other group, often look to technology as a path to innovation and progress.

Technology may offer advanced tools that foster improved operations, create efficiencies,

optimize interpersonal relations, and provide novel solutions to complex issues that arise within

the workplace. During the past several decades, New Yorkers saw the workplace evolve as the

use of fax machines, email, cloud computing, cellular phones, WiFi, GPS tracking, and other

technologies changed and innovated how New Yorkers and the world worked. We now stand

witness to the advent of artificial intelligence in the workplace.

SHRM submits these comments in response to the proposed rules issued by the

Department of Consumer and Worker Protection (“DCWP” or “Department”) on September 23,

2022, that stand to implement New York City Local Law 144 of 2021 (“LL 144”) regarding

automated employment decision tools (“AEDT”). We offer these comments with the insight of

human resources professionals so as to make the proposed rule more effective.

SHRM applauds the Department’s efforts to clarify the obligations of employers and

employment agencies under LL 144. SHRM agrees that clearly defined terms and clarity

regarding the bias audit requirements, the notice requirements, and other obligations under LL

144 serve to promote compliance and reduce confusion about the nuances of this first-of-its-kind

law.

SHRM also commends the Department for its focus on addressing ambiguities and providing

specificity around the requirements and obligations of LL 144 while also refraining from

expanding the scope of the rules to require obligations beyond those that already exist. See

Americans with Disabilities Act of 1990 (ADA), 42 U.S.C. §§ 12101-12213 (2018); NYC

Human Rights Law, N.Y.C. Admin. Code § 8-107. In particular, we recognize the Department’s

efforts to clarify that while employers have certain obligations to reasonably accommodate

workers on the basis of a disability, there is no obligation to provide alternative selection or

evaluation processes. See §§5-303 (c) and (e).

The Equal Employment Opportunity Commission has recently provided valuable guidance to

employers on the issue of automated employment decision tools and the impacts on individuals

with disabilities. See U.S. Equal Employment Opportunity Commission, The Americans with

Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job

Applicants and Employees, 29 CFR Part 1630 & app. (May 12, 2022), available at

https://www.eeoc.gov/laws/guidance/americans-disabilities-act-and-use-software-algorithmsand-

artificial-intelligence. SHRM submits that all NYC employers and employees will stand to

benefit from that guidance even as NYC moves forward with the requirements set forth in LL

144.

While recognizing that the issues are complex and require thoughtful implementation, the

proposed rules require additional clarification. In the sections that follow, we summarize the key

areas where further clarification could assist employers as they work towards implementing LL

144.

A. THE DEFINITION OF AUTOMATED EMPLOYMENT

DECISION TOOL IS OVERLY BROAD AND RISKS UNINTENDED

APPLICATION

SHRM commends the Department on its effort to further clarify the definition of AEDT.

As it stands, LL144 § 20-870 defines AEDT as

any computational process derived from machine learning,

statistical modeling, data analytics, or artificial intelligence that

issues simplified output, including a score, classification, or

recommendation that is used to substantially assist or replace

discretionary decision making for making employment decisions

that impact natural persons. The term “automated employment

decision tool” does not include a tool that does not automate,

support, substantially assist or replace discretionary decisionmaking

processes and that does not materially impact natural persons,

including, but not limited to, a junk email filter, firewall, antivirus

software, calculator, spreadsheet, database, data set, or other

compilation of data.

The Department’s proposed rules, § 5-300, add that “the phrase ‘to substantially assist or

replace discretionary decision making’” as used in LL 144 § 20-870, means to: (1) “rely solely

on a simplified output (score, tag, classification, ranking, etc.), with no other factors considered,”

or (2) “to use a simplified output as one of a set of criteria where the output is weighed more

than any other criterion in the set,” or “to use a simplified output to overrule or modify

conclusions derived from other factors including human decision-making.”

Further, “machine learning, statistical modeling, data analytics, or artificial intelligence,” as used

in LL 144 § 20-870, as defined in the proposed rules, means a group of mathematical, computer-

based techniques:

i. that generate a prediction, meaning an expected

outcome for an observation, such as an assessment

of a candidate’s fit or likelihood of success, or that

generate a classification, meaning an assignment of

an observation to a group, such as categorizations

based on skill sets or aptitude; and

ii. for which a computer at least in part identifies the

inputs, the relative importance placed on those

inputs, and other parameters for the models in order

to improve the accuracy of the prediction or

classification; and

iii. for which the inputs and parameters are refined

through cross-validation or by using training and

testing data.

Further, “simplified output,” as used in LL 144 § 20-870, means a

prediction or classification as specified in the definition for

“machine learning, statistical modeling, data analytics, or artificial

intelligence.” A simplified output may take the form of a score

(e.g., rating a candidate’s estimated technical skills), tag or

categorization (e.g., categorizing a candidate’s resume based on

keywords, assigning a skill or trait to a candidate), recommendation

(e.g., whether a candidate should be given an interview), or ranking

(e.g., arranging a list of candidates based on how well their cover

letters match the job description). It does not refer to the output

from analytical tools that translate or transcribe existing text, e.g.,

convert a resume from a PDF or transcribe a video or audio

interview.

The purpose of this law will be served best where employers have clear guidance as to when the

law is triggered (i.e., which tools, processes, or systems are covered) and what, specifically,

employers need to do to comply. While these further definitions seek to create clarity, SHRM

observes that they nevertheless cast a very wide net as to the employer tools, processes, or

systems that might fall into the definition of AEDT. This ambiguity is likely to lead to employers

being left to guess as to what might or might not be covered by this definition. It is helpful that

the definition excludes a “junk email filter, firewall, antivirus software, calculator, spreadsheet,

database, data set, or other compilation of data,” however the broad and conceptual phrasing of

these definitions insufficiently identifies specific, concrete, reliable criteria for determining what

tools, processes, or systems are covered. We encourage the final regulations to explicitly exclude

automated tools that carry out human-directed assessments.

For example, some employers might use a scheduling tool that captures employee availability for

purposes of both shift scheduling and candidate evaluation. That tool may “generate a

prediction” where “a computer … identifie[d] the inputs” and decided the “relative importance

placed on those inputs … in order to improve the accuracy of the prediction or classification,”

which involved “inputs and parameters” that were “refined through crossvalidation or by using

training and testing data.” And the purpose of this tool may be “to screen candidates for

employment” by virtue of determining and calculating how to best schedule individuals to work.

Moreover, with large candidate pools, employers rely on automation to screen candidates based

on core job-related decision points such as educational attainment or relevant licensure.

Presumably, these types of tools are not intended to be covered by this law, as subjecting this

type of tool to the requirements of LL 144 would likely create a severe burden upon employers,

and the continued use of such tools would be untenable in the face of the requirements and

penalties under LL 144.

Furthermore, nearly 1 in 4 organizations use automation or AI to support HR-related activities,

with 79% of the organizations utilizing AI in recruitment and hiring. Mostly extralarge

organizations with 5,000+ employees (42%) and large employers with 500-4,999 employees

(26%) are utilizing AI to support HR. In the next five years, 1 in 4 organizations plan to start

using or increase their use of AI in the recruitment and hiring process. These tools will become

more essential to communicating with candidates, reviewing resumes, and identifying skills.

These tools will provide more opportunities and employment for untapped talent pools –

veterans, opportunity youth, older workers, individuals living with disabilities, and people with

criminal records.

Clarity and specificity will foster compliance with LL 144, which in turn will promote the goals

and objectives of LL 144. Thus, SHRM recommends further specificity and clarity to assist New

York City employers in compliance with this new law.

B. THE BIAS AUDIT PROVISIONS REQUIRE FURTHER CLARITY

SHRM commends the Department on its effort to provide clarity regarding the criteria for a “bias

audit.” However, additional clarification on the bias audit requirements is needed.

First, the scope of the candidate pool required to be tested is unclear. Specifically, it is not clear

whether employers are required, or conversely, permitted, to conduct a bias audit on all

candidates globally (i.e., in and outside of New York City in the case of multijurisdictional

employers), only New York City candidates, or a sample set of either category. Presumably, the

jurisdictional reach of LL 144 does not extend outside of New York City. Otherwise,

multijurisdictional employers with significant workforces outside of New York City could be

unreasonably burdened by the requirements and obligations of LL 144.

Employers should be expressly permitted to use broad data for the required bias audit purposes.

For instance, § 5-302 requires that “[p]rior to the use of an AEDT, employers and employment

agencies in the city must make … publicly available” “the date of the most recent bias audit of

such AEDT and a summary of the results,” as well as “the distribution date of the AEDT to

which such bias audit applies.” This requirement appears to create a “chicken-or-theegg”

problem. An NYC employer looking to implement a new AEDT (or an updated version of

existing AEDT, as discussed below) would have no prior data upon which to perform a bias

audit, at least not in the first year, and thus could not begin to use the AEDT to obtain such data.

The effect appears to be a de facto prohibition on implementing new AEDT tools. This issue

would be easily addressed if the proposed rules were revised to clarify that NYC employers with

operations outside the jurisdiction of NYC are permitted to include candidate data from other

jurisdictions or alternately to use sample test data to conduct a bias audit of the AEDT to comply

with LL 144.

Second, the proposed rules should clarify the timing implications with regard to the cadence of

any required bias audits. At least one potential issue arises for employers who update or

otherwise improve an AEDT. There should be a good faith standard to assess compliance in

those instances. So long as a bias audit has been conducted on the tool itself, employers should

not be required to conduct a new audit whenever enhancements are introduced. This is

particularly true given that AEDT tools are continuously evolving.

Given that AEDTs are being used in the context of rapidly changing technological

advancements, as well as the potential for penalties even in spite of employers' good faith and

best efforts, SHRM recommends that the Department include a safe-harbor provision that

permits employers to be deemed to have complied with LL 144 based on good faith efforts

toward compliance. The Department should further consider providing that this safe harbor

permits an AEDT to remain in use for the remainder of the time period during which the last bias

audit was conducted.

Third, SHRM respectfully submits that the Impact Ratio metric that relies simply on

“average scores” is a flawed methodology that does not adequately consider the effect size or the

magnitude of the difference in scores. It appears to be a novel approach that, while seemingly

simple to calculate, does not appear to be readily used by statisticians, labor economists, or

industrial-organizational psychologists in the employment context. Indeed, we have been unable

to find any interpretative guidance or peer-reviewed literature to support the use of the proposed

“average score” methodology to determine an impact ratio result.

The use of the “average scores” impact ratio analysis is particularly problematic because it

appears that it will be required in virtually all AEDTs. While two impact ratio methodologies are

defined in the proposed rules and are seemingly designed to be used in the alternative,

it is

difficult to conceive of an AEDT that “selects individuals in the hiring process or classifies

individuals into groups” as set forth in § 5-301(a), without scoring applicants or candidates in

some way as described in § 5-301 (b). Accordingly, under the proposed rules, NYC employers

will be required to conduct a bias audit based on a flawed methodology that lacks an adequate

benchmark for assessment purposes.

Fourth, while SHRM generally supports transparency efforts, there is some concern that the

required notices could risk disclosure of candidates’ personal identifiable information (“PII”) or

Impact Ratio. “Impact ratio” means either (1) the selection rate for a category divided by the selection rate of the most selected

category or (2) the average score of all individuals in a category divided by the average score of individuals in the highest scoring

category.

other information that is otherwise protected from disclosure. It is presumed that neither LL 144

nor the proposed rules require or otherwise seek disclosure of such information. SHRM thus

recommends that the Department consider providing clarification in § 5-302 that “Nothing in this

subchapter requires an employer or employment agency to disclose any personal identifiable

information or any other protected or confidential information of any candidate or employee.”

Finally, SHRM commends the Department on the flexibility it has offered employers in

determining who is considered to be an “independent auditor” of AEDTs. As NYC employers

move quickly to come into compliance, being able to rely on a broad range of individuals or

companies to implement the appropriate audit mechanisms will be of critical importance.

C. THE DEPARTMENT SHOULD GUARD AGAINST THE DANGERS OF

STIFLING INNOVATION

SHRM commends the Department on its efforts to propose rules to create protections for

candidates while providing clear, workable obligations for employers in this developing area

where technology and law intersect.

The development and enhancement of advanced technologies in the workplace require flexibility

to ensure that optimal processes benefit both the employer community, and candidates and

employees in New York City. The war for talent, particularly after the pandemic, has created

unique challenges for employers, including increased complexities due to the ever-evolving

remote workforce.

As employers continue to navigate these real challenges, employers are exploring the use of

AEDTs in myriad ways, including quickly sourcing and expanding candidate pools, identifying

skills gaps, enhancing talent pipelines, reducing turnover, increasing productivity, and bolstering

diversity equity and inclusion goals. Nearly 80% of employers report that candidates that score

highly on a pre-employment assessment but do not meet the minimum experience requirement

are likely to advance in the application and hiring process. Nearly 70% report the same for

candidates that fail to meet the minimum education requirements for a position and are more

likely to advance with high skills assessments. When carefully developed and implemented,

employers and workers alike experience the positive benefits of AEDTs. Employers experience

improved efficiencies which in turn benefits workers by optimizing the selection of candidates to

ultimately place them in jobs that provide increased fulfillment and opportunities for success

while enhancing career progression and movement in the workforce.

In light of the key benefits that AEDTs offer, SHRM submits that the requirements and

obligations contained in the proposed rules should be viewed through the lens of minimizing

limitations on the growth and advancements that could benefit everyone. SHRM champions the

creation of better economic opportunities for overlooked and untapped talent pools, including

veterans, older workers, individuals living with disabilities, opportunity youth, and people with a

criminal record. The availability of AEDTs to better recognize the knowledge, skills, and

abilities of these workers provides an important tool for organizations seeking to build a more

equitable and inclusive workplace. SHRM promotes the value of untapped talent as more than a

matter of social responsibility or goodwill; these groups of workers are proven to show high

returns on investment and skills for employers. Appropriate safeguards must be balanced against

heavy-handed regulatory restrictions that will set key HR functions back and impede the ability

to create and identify broader, more inclusive talent pipelines. To do otherwise will only dampen

innovation and stifle creativity within the confines of employment for NYC workers.

SHRM commends the Department on its efforts to propose rules that pioneer the realm of

artificial intelligence and machine learning, making these rules among the first of their kind. We

appreciate the opportunity to present our views on the proposed rules and would welcome the

opportunity to assist the Department as it continues its rule-making process. If you should have

any questions or require any additional information, please feel free to contact James Redstone at

[email protected] or C. Mitch Taylor at Mitc[email protected].

Sincerely,

Emily M. Dickens

Chief of Staff & Head of Government Affairs

Testimony of Julia Stoyanovich before the New York City Department of

Consumer and Worker Protection regarding Local Law 144 of 2021 in Relation to

Automated Employment Decision Tools (AEDTs)

October 23, 2022

Dear Chair and members of the Department:

My name is Julia Stoyanovich. I hold a Ph.D. in Computer Science from Columbia University in

the City of New York. I am an Associate Professor of Computer Science and Engineering at the

Tandon School of Engineering, an Associate Professor of Data Science at the Center for Data

Science, and the founding Director of the Center for Responsible AI at New York University. In

my research and public engagement activities, I focus on incorporating legal requirements and

ethical norms, including fairness, accountability, transparency, and data protection, into data-

driven algorithmic decision making.

I teach responsible data science courses to graduate and

undergraduate students at NYU.

Most importantly, I am a devoted and proud New Yorker.

I would like to commend New York City on taking on the ambitious task of overseeing the use

of automated decision systems in hiring. I see Local Law 144 as an incredible opportunity for

the City to lead by example, but only if this law is enacted in a way that is responsive to the

needs of all key stakeholders. The conversation thus far has been dominated by the voices of

commercial entities, especially by AEDT vendors and organizations that represent them, but also

by employers who use AEDT, and by commercial entities wishing to conduct AEDT audits.

However, as is evident from the fact that we are testifying in front of the Department of

Consumer and Worker Protection, the main stakeholder group Local Law 144 aims to protect –

from unlawful discrimination, and arbitrary and capricious decision-making – are job candidates

and employees. And yet, their voices and the voices of their representatives are conspicuously

missing from the conversation!

As an academic and an individual with no commercial interests in AEDT development, use, or

auditing, I am making my best effort to speak today to represent the interests of the job

candidates, employees, and the broader public. However, I cannot speak on behalf of this

diverse group alone. Therefore, my main recommendation today is that New York City must

ensure active participation of a diverse group of job seekers, employees and their representatives

in both rule making and enactment of Local Law 144.

For background: I actively participated in the deliberations leading up to the adoption of Local

Law 144 of 2021

and have carried out several public engagement activities around this law

See https://dataresponsibly.github.io/ for information about this work, funded by the National Science Foundation through NSF

Awards #1926250, 1934464, and 1922658.

All course materials are publicly available at https://dataresponsibly.github.io/courses/

Testimony of Julia Stoyanovich before New York City Council Committee on Technology regarding Int

-2020, November 12, 2020, available at

https://dataresponsibly.github.io/documents/Stoyanovich_Int1894Testimony.pdf

when it was proposed

. Informed by my research and by opinions of members of the public, I

have written extensively on the auditing and disclosure requirements of this Law, including an

opinion article in the New York Times

and an article in the Wall Street Journal

. I have also

been teaching members of the public about the impacts of AI and about its use in hiring, most

recently by offering a free in-person course at the Queens Public Library called “We are AI”

(Course materials are available online

.) Based on my background and experience, I would like

to make 4 recommendations regarding the enforcement of Local Law 144.

Recommendation 1: Involve job seekers, employees, and their representatives in

defining standards for AEDT audits and notices.

The NYU Center for Responsible AI (R/AI) conducted numerous public engagement activities

under my leadership, both broadly on AI and automated decision making, and specifically on

AEDTs, and we see substantial interest from members of the public. R/AI will be happy to

assist the City in convening diverse groups of stakeholders.

Recommendation 2: Expand the scope of auditing for bias beyond disparate impact to

include other dimensions of discrimination, based on input from all key stakeholders,

including job seekers, employees, and their representatives.

For example, the most prominent thread in readers’ comments on a New York Times opinion

piece I co-authored in March 2021 concerned age-based discrimination in hiring. Local Law 144

does not currently control include any provisions to audit for this type of discrimination.

Recommendation 3: Expand the scope of auditing beyond bias to also interrogate

whether the AEDTs work, based on input from all key stakeholders, including job

seekers, employees, and their representatives.

There is evidence to suggest that recommendations of many of these tools are inconsistent and

arbitrary

. AEDTs that don’t work hurt job seekers and employees, subjecting them to arbitrary

decision-making with no recourse. AEDTs that don’t work also hurt employers, they waste

Public engagement showreel, Int 1894, NYU Center for Responsible AI, December 15, 2022 available at

https://dataresponsibly.github.io/documents/Bill1894Showreel.pdf

We need laws to take on racism and sexism in hiring technology, Alexandra Reeve Givens, Hilke

Schellmann and Julia Stoyanovich, The New York Times, March 17, 2021, available at

https://www.nytimes.com/2021/03/17/opinion/ai-employment-bias-nyc.html

Hiring and AI: Let job candidates know why they were rejected, Julia Stoyanovich, The Wall Street Journal Reports:

Leadership, September 22, 2021, available at https://www.wsj.com/articles/hiring-job-candidates-ai-11632244313

“We are AI” series by NYU Tandon Center for Responsible AI and Queens Public Library helps citizens take control of tech,

March 14 2022, available at

https://engineering.nyu.edu/news/we-are-ai-series-nyu-tandon-center-responsible-ai-queens-public-library

“We are AI: Taking control of technology”, NYU Center for Responsible AI, available https://dataresponsibly.github.io/we-are-

ai/

“Resume Format, LinkedIn URLs and Other Unexpected Influences on AI Personality Prediction in

Hiring: Results of an Audit,” Rhea et al., AAAI/ACM AIES 2022, available at https://dl.acm.org/doi/10.1145/3514094.3534189

money paying for software that doesn't work, and miss out on many well-qualified candidates

based on a self-fulfilling prophecy delivered by a tool.

In my own work, done in collaboration with an interdisciplinary team that included several data

scientists, a sociologist, an industrial-organizational (I-O) psychologist, and an investigative

journalist, I evaluated the validity of two algorithmic personality tests: AEDTs that are used for

pre-employment assessment

, Humantic AI and Crystal. Importantly, rather than challenging or

affirming the assumptions made in psychometric testing — that personality traits are meaningful

and measurable constructs, and that they are indicative of future success on the job— we framed

our methodology around testing the assumptions made by the vendors themselves.

We found that both systems show substantial instability on key facets of measurement, and so

cannot be considered valid testing instruments. For example, Crystal frequently computes

different personality profiles if the same resume is given in PDF vs. in raw text, while Humantic

AI gives different personality profiles on a LinkedIn profile vs. a resume of the same job seeker,

violating the assumption that the output of a personality test is stable across job-irrelevant input

variations. Results are summarized in Table 1.

Table 1: Summary of stability results for Crystal and Humantic AI:

✔

indicates sufficient rank-

order stability in all traits, while

✗

indicates insufficient rank-order stability or significant

locational instability in at least one trait, and N/A indicates the facet was not tested in our audit.

Results are detailed in https://dl.acm.org/doi/10.1145/3514094.3534189.

Recommendation 4: Disclose information about job qualifications and characteristics

for which the AEDT screens in a manner that is comprehensive, specific, understandable,

and actionable for job seekers and employees.

I recommend showing job seekers and employees simple, standardized labels that list the factors

that go into the AEDT’s decision both before they are screened and after a decision is made. Job

seekers, employees, and their representatives should be directly involved in the design and

testing of such labels.

Figure 1 gives an example of a possible “posting label” with a short and clear summary of the

screening process. (See my recent Wall Street Journal article for details

.) This label is presented

to a job seeker before they apply, supporting informed consent, allowing them to opt out of

components of the process, or to request accommodations. Giving an opportunity to request

accommodations is particularly important in light of the recent guidance by the Equal

Employment Opportunity Commission on the Americans with Disabilities Act and the use of AI

to assess job applicants and employees

Figure 1: A posting label is a short, simple, and clear summary of the screening process. This

label is presented to a job seeker before they apply, supporting informed consent, allowing them

to opt out of components of the process or to request accommodations.

I would like to keep my testimony today brief. I am enclosing a copy of the testimony I entered

on June 6, 2022, for additional background on Automated hiring systems, and for details

regarding my recommendations on rules for auditing and notice (disclosure) requirements of

Local Law 144.

Testimony of Julia Stoyanovich before the New York City Department of

Consumer and Worker Protection regarding Local Law 144 of 2021 in Relation to

Automated Employment Decision Tools

June 6, 2022

Dear Chair and members of the Department:

My name is Julia Stoyanovich. I hold a Ph.D. in Computer Science from Columbia University. I

am an Associate Professor of Computer Science and Engineering at the Tandon School of

Engineering, and an Associate Professor of Data Science at the Center for Data Science, and the

founding Director of the Center for Responsible AI at New York University. In my research and

The Americans with Disabilities Act and the use of software, algorithms, and AI to assess job applicants and employees, US

Equal Employment Opportunity Commission, 2022,

https://www.eeoc.gov/laws/guidance/americans-disabilities-act-and-use-software-algorithms-and-artificialintelligence

public engagement activities, I focus on incorporating legal requirements and ethical norms,

including fairness, accountability, transparency, and data protection, into data-driven algorithmic

decision making.

I teach responsible data science courses to graduate and undergraduate

students at NYU.

Most importantly, I am a devoted and proud New Yorker.

I actively participated in the deliberations leading up to the adoption of Local Law 144 of

2021

and have carried out several public engagement activities around this law when it was

proposed

. Informed by my research and by opinions of members of the public, I have written

extensively on the auditing and disclosure requirements of this Law, including an opinion article

in the New York Times

and an article in the Wall Street Journal

. I have also been teaching

members of the public about the impacts of AI and about its use in hiring, most recently by

offering a free in-person course at the Queens Public Library called “We are AI”

. Course

materials are available online

In my statement today I would like to make three recommendations regarding the

enforcement of Local Law 144 of 2021:

1. Auditing: The scope of auditing for bias should be expanded beyond disparate impact to

include other dimensions of discrimination, and also contain information about a tool’s

effectiveness - about whether a tool works. Audits should be based on a set of uniform

publicly available criteria.

2. Disclosure: Information about job qualifications or characteristics for which the tool

screens the job seeker should be disclosed to them in a manner that is comprehensible

and actionable. Specifically, job seekers should see simple, standardized labels that show

the factors that go into the AI’s decision both before they apply and after a

decision on their application is made.