The AI Screening Evidence Trail: Audit, Notice, Log, Renew

You run an AI tool that screens or ranks job candidates, and you are the person who has to answer for how it was cleared to run. This guide is for recruiting operations, revenue operations, and anyone else accountable for the data and the method behind an automated screen. It gives you the end-to-end evidence trail - independent bias audit, public summary, timed candidate notice, offered alternative, and per-decision log - assembled in order across NYC, Illinois, Colorado, California, and the EU deployer role, with the artifacts, timing windows, and record rows you keep renewed on cycle.

Most pages on this subject explain one law in isolation, usually NYC Local Law 144, or they sell you an audit. Neither assembles the whole trail for an owner whose applicants sit in several states and the EU. That is the gap this fills.

What the evidence trail actually is

The evidence trail is five artifacts that together prove your AI screening tool was lawful before it touched a single application: an independent bias audit, a published summary of it, a timed candidate notice, a documented path to an alternative process, and a per-decision log that lets the next audit recompute everything. The core insight to hold onto is that the audit is a calculation, not a document. A tidy PDF that says the tool "showed no significant bias" proves nothing, because compliance lives in the log that feeds the rates and ratios, and a stale summary without a maintained log cannot be re-audited when the 12-month clock turns.

Everything downstream depends on getting the vocabulary straight first. The same tool wears a different legal label in each place it operates.

A term of art defined once: an AEDT, or automated employment decision tool, is the NYC label for software that substantially assists or replaces a hiring decision. California calls its version an ADS (automated decision system) under the FEHA regulations, the California Privacy Protection Agency calls its own version an ADMT (automated decision-making technology), Illinois regulates "covered AI" under its Human Rights Act amendment, and the EU classifies employment screening as an Annex III high-risk system. One tool, five labels, five slightly different duties.

Which jurisdictions impose what

The US duties are not uniform: only NYC currently requires a recurring independent bias audit with a published summary, while Illinois, Colorado, and California lean on notice, disclosure, and anti-discrimination testing instead. Build the audit for NYC and let it do double duty as evidence elsewhere, because a clean impact-ratio analysis is exactly the testing California's FEHA rules treat as evidence of non-discrimination.

Here is the US obligation matrix, drawn straight from the dossier's dates and duties.

Two dates on that table are moving targets you should re-check rather than memorize. Illinois signed HB 3773 on August 9, 2024, effective January 1, 2026; its Department of Human Rights published proposed implementing rules on May 15, 2026 and withdrew them on June 2, 2026, which did not pause the law. Colorado repealed its original AI act and replaced it with SB 26-189, now effective January 1, 2027, imposing narrower notice and disclosure duties. When a rulemaking is in flux, track the statute's effective date and the agency's docket, not a blog's snapshot.

The EU timeline moved too, and in a way that rewards reading the fine print.

The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 and pushed Annex III high-risk deployer duties from 2 August 2026 to 2 December 2027. It did not move Article 50 transparency, which still starts on 2 August 2026. Treat the delay as a build window, not a reprieve: the standards simply were not ready, and Article 26's log-retention and human-oversight duties are the parts that take real time to wire.

What the bias audit must compute

An independent auditor computes, from your applicant data, the selection rate for each sex category, each race and ethnicity category under the EEO-1 framework, and each intersectional sex-by-race category, then divides each group's rate by the most-selected group's rate to produce an impact ratio. Where the tool outputs a score rather than a pass or fail, the auditor computes scoring rates against the full-sample median instead. This is the heart of an AEDT bias audit: a set of ratios, each one a focal group's rate over the top group's rate.

The four-fifths rule sets the trigger line at 0.80. It comes from EEOC adverse impact analysis: if any demographic group's selection rate is less than 80% of the highest-selected group's rate, that may signal adverse impact against that group. Worked plainly, if men are selected at 60% and women at 40%, the impact ratio for women is 40/60 = 0.67, which sits below the 0.80 line and flags for review.

Aggregate ratios that look clean can hide a failing cell. A tool can be balanced on sex and balanced on race while disadvantaging one specific sex-by-race group, which is exactly why Local Law 144 forces the cross-tabulation. In one real published summary, a top-line group cleared while a Female, Two or More Races cell came in at 0.87. Read every intersectional cell, not just sex and race separately.

The published summary is not a slogan. It must carry the source and explanation of the data used, the date of the last audit, the number of applicants or candidates, the selection or scoring rates, the impact ratio by race/ethnicity, sex, and intersectional categories, and the number of individuals in an unknown category. Auditors may exclude a category under 2% of the data if they state a justification. A page that only says the tool "was audited and showed no significant bias" does not meet the standard.

Choosing a genuinely independent auditor

An independent auditor is a person or organization that has not been involved in using, developing, or distributing the tool, has no employment relationship with you or the vendor, and holds no direct or material indirect financial interest with either party. DCWP maintains no approved list, so selecting a qualified, genuinely independent auditor is your responsibility, not a box someone else ticks for you. The most common conflict is an "auditor" who also consults on the same tool - that fails independence no matter how good the statistics are.

The talent to run this work is scarce and geographically lopsided, which is a large part of why employers lean on external auditors in the first place.

When you go looking for the auditor or the internal owner, the filter that matters is independence plus demonstrated impact-ratio experience, not a generic data-science background. Describing that person in plain language is faster than reverse-engineering it from a directory.

Refolk is where I turn that independence test into a search instead of a stack of LinkedIn tabs, because the disqualifying signal - a vendor affiliation - is the kind of thing you want surfaced before you sign an engagement, not after.

The build-and-maintain procedure

Run these eight steps in order. The first two scope the work, the middle four build the trail, and the last two cover the EU layer and the renewal clock that keeps the whole thing valid.

Rough timing: the inventory and classification take two to three weeks of mostly legal and ops time, the audit runs three to six weeks once the auditor has data, and the notice and log plumbing take two to four weeks each in the ATS and data warehouse. The EU layer is ongoing work to the December 2027 deadline, and the renewal clock is a standing annual commitment.

The 10-business-day notice and the alternative

Candidates who are assessed in NYC must receive an individual notice at least 10 business days before the tool evaluates them, and that notice must state that an AEDT will be used, name the job qualifications and characteristics the tool will assess, and tell candidates how to request an alternative selection process or an accommodation. The single most common defect is counting calendar days: 10 calendar days is only about 7 business days, which is short of the rule.

Note the limit of the obligation. The notice tells candidates they may request an alternative selection process. The law does not force you to provide one. You decide your own response policy, but you must inform and you must capture the request against the candidate record.

Subject: How your application for [role, location] will be assessed

We use an automated employment decision tool to help assess applications for this role.

What the tool assesses: [job qualifications and characteristics, e.g. relevant experience, skills match].

You can read our bias audit summary here: [link to published summary].

You may request an alternative selection process or a reasonable accommodation. To do so, reply to this message or contact [named owner / inbox] by [date].

The automated assessment will not begin before [date = at least 10 business days from send, business days only].

Ten calendar days is seven business days with a weekend in it, and seven is a violation waiting to be found.

The per-decision log that keeps you auditable

The per-decision log is the spine of the whole trail: it is the record from which every future audit recomputes selection rates and impact ratios, so without it you hold a one-time PDF that cannot be renewed. One row per candidate, written at the moment of screening, is what makes next year's audit a recalculation rather than a reconstruction.

For impact ratios to be computable, each row must retain the candidate's EEO-1 sex and race/ethnicity category, the selection outcome or the score plus whether it beats the full-sample median, the notice date, and any alternative request, and the table as a whole must carry a count of unknown-category individuals and any sub-2% category the auditor excluded. Miss the category fields and the audit cannot cross-tabulate; miss the notice date and you cannot prove timing per candidate.

EU deployer duties without over-scoping

If your tool touches EU applicants, you are a deployer of an Annex III high-risk system and you owe the Article 26 duties: use the system in accordance with instructions, assign human oversight to named people, monitor it, maintain data input quality, and retain the system logs. What most private employers do not owe is an Article 27 fundamental rights impact assessment. Take a Belgian SME that buys a tool to screen job applications: the system is high-risk, Annex III point 4 applies, and Article 26 applies in full, but Article 27 does not, because the company is not a public body, provides no public service, and CV screening is neither prohibited practice 5(b) nor 5(c).

This distinction carries money. A completed FRIA triggers a mandatory notification to the market surveillance authority in all cases - unlike a GDPR DPIA, the notification is not conditional. Misclassifying yourself as in-scope manufactures a regulatory touchpoint you did not need. Confirm your public-body or public-service status before you scope a FRIA, not after.

How this goes wrong

The trail fails in predictable places, and most failures are false positives - a dashboard that says "done" while the underlying duty is unmet. These are the ones worth auditing yourself against, because thin enforcement does not mean low exposure. A December 2025 New York State Comptroller audit found DCWP's record thin: only 2 complaints and 1 instance of noncompliance across 32 companies over the law's first two years, versus at least 17 instances of potential noncompliance the Comptroller's own auditors found among the same firms. A documented "ineffective" finding plus committed reforms means the baseline going forward is heightened scrutiny of exactly these gaps.

• Calendar days masquerading as business days. The dashboard shows "10 days' notice sent" but counts calendar days, roughly 7 business days, short of the rule. Check by recomputing the gap excluding weekends and holidays, per candidate.
• The vanity summary. A page that says the tool "showed no significant bias" with no numbers. Check that the page carries actual rates, impact ratios, applicant count, and unknown and excluded counts.
• Clean aggregate, failing intersection. The tool looks balanced on sex and on race while one sex-by-race cell sits below 0.80. Check every intersectional cell, not the two top-line splits.
• Auditor conflict. The "auditor" also consults on the same tool and so is not independent. Check for any development or distribution role and any material financial interest before you engage.
• Assuming a bought product covers you. The duty applies to you before use, and buying an audited product does not transfer your responsibility. Check that the audit covers your deployed configuration and your applicant pool.
• Audit posted, notice skipped. The summary is live but the talent team never wired the notice, so candidates get screened with no notice at all - each one a separate exposure even though the audit is valid. Check the notice log against every screened requisition.
• The stale audit. Using the tool past the 12-month mark is a violation even if nothing changed. Check with a renewal alarm plus a material-change trigger.
• Mis-scoped FRIA. A private deployer treating itself as Article 27 in-scope. Check public-body or public-service status before scoping anything beyond Article 26.

On the one genuine ambiguity: sources disagree on whether a material retrain or configuration change mandates a fresh audit before the 12-month mark. One treats it as good practice rather than a statutory requirement; another says a material change typically warrants a fresh audit. I would re-audit on any material change rather than argue the point, because the downside of an unneeded audit is cost and the downside of skipping one is a daily-accruing violation. Check your own vendor's change log against your last audit date and decide with legal.

Keeping it current

The trail is not a project you finish; it is a cycle you run. The renewal clock is the artifact that keeps every other artifact valid, because an audit older than one year is itself a violation and a summary pulled too early leaves a gap. Set the re-audit alarm for 11 months, not 12, so you have slack, and keep the published summary live at least six months after the tool's last use.

When you re-audit, you are not starting over. The per-decision log you maintained all year is the dataset, the summary template is written, and the notice already fires. Re-running the cycle is a recalculation and a republish. That is the payoff of treating the audit as a calculation rather than a certificate: the work you did to make the first audit defensible is exactly the work that makes every renewal cheap.