The Candidate Database Cleanup and Deduplication Playbook

This playbook is for the recruiting operations or revenue operations owner accountable for a candidate database: the person who has to explain why pipeline counts are inflated, why a search misses people who are clearly in the system, and how the data was gathered when legal asks. It gives you the ordered procedure to run a full cleanup this week - measure the real duplicate rate, merge duplicates without false positives, purge records past their retention window, standardize fields, and set an audit cadence that keeps the database clean. Every threshold, merge rule, and purge trigger here is one you can act on at the keyboard.

Most public advice stops at "regularly review your data." That is not a method. What follows is.

Why a dirty candidate database costs you more than storage

A dirty candidate database quietly breaks every number you report and every search you run, and the damage compounds because duplicates hide the very history that would have caught them. This is not a cosmetic problem. It is a measurement problem and a compliance problem at the same time.

Duplicate records average 15 to 20% of all data in an organization, per Experian's data-hygiene figures. For candidate databases specifically, no single authoritative rate is published, but one practitioner guide estimates most teams underestimate the scale of the problem by 3 to 4x. That gap matters because you cannot manage what you have mismeasured.

The inflation shows up in three places at once. Pipeline analytics overcount unique candidates. Sourcing reports credit recruiters for "new" talent that was already in the system. And hiring managers do not see relevant feedback from previous interview cycles because it sits on a duplicate profile they never open. A candidate rejected for a culture reason last quarter reappears as a fresh lead this quarter, and nobody knows.

There is a real ceiling to aim at. Industry references put the achievable duplicate-rate benchmark at roughly 1% of records, and report that only about 22% of organizations actually hit that target. So the target is 1%, the median is worse than you think, and the fix is a repeatable procedure rather than a purchase.

Who owns this work is worth naming, because it is thinner than most people assume. In Refolk's index of professional profiles, 102 people in the United States hold the exact title "Recruiting Operations," at employers including Anthropic, Sierra, OpenAI, Plaid, and honeycomb.io. Compare that to 415 US professionals for "Revenue Operations" - about four times the headcount for the function that owns the same class of data problem in the sales stack.

Column source: counts from Refolk's index; ratio derived (415 divided by 102).

The point of that table is not to complain about staffing. It is that a candidate database cleanup usually lands on one person, sometimes a fraction of one person, and the procedure has to be efficient enough to survive that. The rest of this guide assumes you are that person.

What causes duplicate candidate records in the first place

Duplicates come from three mechanisms, and knowing which one dominates your database tells you where to aim prevention. The three are: candidates applying through multiple channels, candidates using different email addresses across applications, and recruiters manually re-entering someone who was already sourced.

Each mechanism defeats a different detection strategy. A candidate who applies twice with the same email is caught by an exact-email match. A candidate who applies through a job board and then through your careers site with a personal Gmail one time and a work address the next is not. A recruiter who sources someone from the open web, does not check, and creates a new record has produced a duplicate with a completely different data shape from the original.

The single most useful thing to understand before you start: systems relying only on exact email matching miss most of these cases. If your ATS shows a low duplicate count, that is often the report lying to you, not a clean database. The test is cheap - rerun detection with fuzzy name plus phone on a sample and see how many the exact-email report hid.

Contact data also decays, which turns yesterday's clean record into today's near-duplicate. Work email decays 20 to 30% annually, job title 15 to 25%, and direct phone 15 to 20%. In high-turnover sectors like staffing and tech, overall B2B contact data decays around 70% per year. So even a database that was clean at import drifts toward duplication and staleness on its own. That is the argument against one-off cleaning, and I will return to it.

How to set matching rules and thresholds that merge without false positives

Set your matching rules as composite scores across multiple fields, then apply a confidence threshold, and reserve a manual-review band for the scores in between. Single-field exact matching is too blunt in both directions: it misses reapplicants and, when you loosen it, it merges different people.

The mechanism is straightforward. A similarity algorithm scores how closely two values resemble each other - for example how close "T. Gupta" is to "Tejus Gupta." You compute that per field, combine the field scores into a composite match confidence, and then decide by threshold. The whole art is where you set the threshold and what you do with the uncertain middle.

The threshold trade-off is documented and unavoidable. At an aggressive threshold around 0.75 you catch more duplicates but risk false merges. At a conservative threshold around 0.90 you avoid false merges but miss most semantic duplicates like "T. Gupta" to "Tejus Gupta." Microsoft's documented tiers name the same trade in plainer terms: low precision is a 30% match, medium is 60%, high is 80%.

My recommendation for a candidate database: set the auto-merge band high, around 0.90, so that everything above it merges without a human. Set a manual-review band between roughly 0.75 and 0.90. Below 0.75, leave records separate. A concrete implementation of this logic - blocking on name, then Levenshtein distance at a threshold on first name with exact match on surname, date of birth, and email - is exactly how the open-source linkage library Splink is used. The same tool can link around one million records on a modern laptop in under two minutes, so scale is not the constraint. Discipline about the review band is.

The review band is the whole safety mechanism. Two different people with a common name in the same city will score high on name and location and land right in the middle. That is the false positive you cannot undo, so it must go to a person.

How to resolve a merge without losing interview history or source attribution

Resolve every merge by survivorship: choose one primary record as the survivor, consolidate the other record's data into it, and archive rather than delete the loser. The goal is one accurate record per candidate, not fewer records overall.

Consolidate, do not discard. When two records represent the same person, move activity history, notes, resume versions, interview feedback, and source attribution into the survivor rather than deleting the older profile outright. ATS merge tooling documents what moves in a merge - notes, contacts, job postings, files, CVs sent, custom templates, and activity records shift from the secondary record to the primary, and the secondary is archived or deleted depending on record type.

Choosing the survivor is a judgement, not a default. A newer record most likely holds more up-to-date contact details, so it is often the logical survivor. But if the existing record already carries job placements or interview history, you may prefer to keep that one and merge the fresh contact details into it. This is why "keep newest" as a blanket rule is dangerous: it silently overwrites the exact history that made the record valuable.

Practitioners disagree on batching, and the disagreement is worth respecting. Some merge all high-confidence pairs in bulk first, then work down. Others say never bulk-merge candidates who have placements. Both are right for their case. Resolve it with the matrix above: bulk-merge the high-confidence, no-history quadrant freely, and hand every record with placement or interview history to a person regardless of confidence.

This is where Refolk earns its place in the procedure. Deduplication logic transfers across the sales and recruiting stacks, and the people who have already made these merge-decision calls are findable by describing the work in plain English rather than guessing at titles.

Run the cleanup: the eight-step procedure

Run the cleanup in this fixed order. The order matters: baseline before you touch anything, segment before you match, and standardize only after merges so you are not standardizing records you are about to consolidate.

Timing, roughly, for a database in the tens of thousands: baseline is half a day to a day for the RecOps owner and ATS admin. Segmentation is half a day for an analyst. Detection is one to three days depending on size. Merge resolution is ongoing, batched by confidence band. The retention purge is one to two days and must include your DPO or legal. Standardization is one to two days for the admin. Prevention controls take about a day. The cadence setup is a one-time hour and then a standing calendar entry.

Purge, archive, or anonymize: retention by jurisdiction

Route every purge decision through the record's jurisdiction and last-contact date, and prefer archive over delete for records still holding hiring value inside their lawful window. Retention is where a cleanup crosses from data hygiene into compliance, and the wrong default in either direction carries cost.

The two failure directions are over-retention and premature deletion. Hold a rejected applicant's file too long and you carry fine risk. Delete it too early and you lose the evidence you would need to defend a discrimination claim. Both are avoidable with a per-jurisdiction rule.

Column source: window and basis columns from ICO, warnergoodman.co.uk, yena.ai, lexgo.be, and talenthr.io.

Read those windows as minimums for defensibility, not maximums for convenience. In the UK an unsuccessful applicant who believes they were discriminated against must bring a claim within six months, so the ICO's retention schedule sets six months from last contact for unsuccessful-applicant records. France under CNIL allows retention in the active database for up to two years from the last contact, provided the profile stays of interest and the candidate has not objected. CNIL separately recommends a five-year intermediate archive purely as discrimination-litigation evidence, which is distinct from keeping someone active in your sourcing pool.

For consented talent pools, 12 to 24 months is the widely accepted range, and beyond 24 months without re-contact or renewed consent, retention is very hard to defend with any European regulator. The stakes are not hypothetical: the GDPR maximum fine is 20 million euros or 4% of worldwide annual revenue, and CNIL fined a real-estate administrator 400,000 euros after candidate documents were exposed.

The operational rule is simple to state and easy to skip: archive is the default for in-window records with hiring value, and deletion happens only when the retention rule fires. Once data is no longer necessary, anonymize or delete it - and log every action so the purge itself is auditable.

Standardize fields so a search surfaces every relevant person

Standardize source, location, title, and disposition fields against controlled picklists, because uncontrolled free text is what makes a search miss people who are demonstrably in the database. A merged, purged database still fails you if the fields are a mess of variants.

The failure is easy to see once you look. Locations arrive as "NYC," "New York," "New York City," and "New York, NY." Source fields carry "LinkedIn," "Linked In," "linkedin," and "LinkedIn Recruiter" as separate values. Phone numbers land in three formats. A search for New York candidates then returns a fraction of the real ones, and a source-of-hire report needs someone to manually combine fifteen variations of "LinkedIn" before it means anything.

The fixes are specific:

• Standardize titles and skills during the save operation, not in a later cleanup.
• Enforce controlled picklists for contract type and location so free text cannot enter.
• Force standardized disposition codes and reason codes when a candidate advances or is rejected.
• Collapse source to a single taxonomy: inbound, referral, agency, LinkedIn outbound, other.

Disposition reasons are load-bearing and the most commonly skipped. Without standardized disqualification reasons, one manager tags "Not a Fit," another "Not Qualified," and a third "Not a Skills Match" - all meaning the same thing, all treated as separate categories, and none of them aggregatable. That is a reporting layer you cannot build after the fact without redoing the tagging.

inbound        - candidate applied directly (careers site, job board)
referral       - introduced by an employee or network contact
agency         - submitted by an external staffing partner
outbound       - sourced and contacted by a recruiter (all channels, incl. LinkedIn)
other          - event, rehire, or anything that does not fit above

Keep the taxonomy short. A source list with twenty entries re-creates the variant problem under a different name. Five values you can defend in a report beat twenty you cannot.

How this goes wrong: the failure modes to guard against

Most cleanups fail in one of eight known ways, and each has a false-positive or false-negative signature you can test for before it costs you. Treat this section as the pre-flight review, because these are the errors that survive a cleanup and surface months later.

• Exact-email-only detection (false negative). The report shows few duplicates, so the database looks clean, but alternate-email reapplicants stay hidden. Test by rerunning with fuzzy name plus phone on a sample and counting what the email report missed.
• Aggressive fuzzy threshold (false positive merge). Two different people with a common name in the same city get merged, and the merge cannot be undone. Guard with a manual-review band; never auto-merge the uncertain middle.
• "Keep newest" survivorship (silent history loss). The survivor looks complete, but placement or interview history from the older record was overwritten. Guard by keeping placed candidates in their own segment so they never auto-merge.
• Flat CSV export for cleanup (severed relationships). Flattening linked objects into CSV rows permanently severs candidate-to-application-to-stage links, producing broken analytics that surface later. Validate those links after any re-import.
• Delete instead of archive (irreversible loss of in-window candidates). A still-valuable candidate inside their retention window is gone for good. Route deletion only through the retention rule; archive everything else.
• Retention purge without jurisdiction logic (fine or lost defense). A blanket purge either over-retains, risking a fine, or deletes too early and loses litigation evidence. Check each record's country and last-contact date against the retention table, and note CNIL's five-year archive.
• Standardization skipped (unusable reports). The source-of-hire report needs manual variant-merging to be accurate. Confirm picklist enforcement is on at save, not optional.
• One-off cleanup with no cadence (re-dirties fast). At around 70% annual contact decay, a scrubbed database re-dirties within months. Confirm a weekly duplicate-creation-rate metric exists before you call the job done.

A completed merge cannot be undone, so the manual-review band is not caution - it is the only safety mechanism you get.

Keep it clean: audit cadence and target metrics

Keep the database clean with a standing control, not an annual scrub: run a full audit quarterly and watch a handful of metrics weekly or monthly on a dashboard. The field has moved from periodic cleaning to continuous control for one reason, and it is decay.

With around 70% annual contact decay in staffing and tech, a once-a-year scrub leaves the database dirty for most of the year between cleanups. So the cadence has two layers. Full comprehensive audits happen at least twice a year, ideally quarterly. Key metrics - duplicate creation rate, email bounce rate, and field completion - are monitored weekly or monthly through a dashboard so you catch drift as it happens rather than at the next audit.

Set explicit targets so the dashboard means something:

• Duplicate rate: aim for roughly 1%; only about 22% of organizations hit it, so treat it as ambitious.
• Accuracy: target 95% or higher; 90 to 94% is acceptable but flags room to improve; below 90% is critical and makes forecasts unreliable.
• Email bounce: below 2%.
• Duplicate creation rate: watched weekly, because a rising number means your intake control has a gap.

The prevention control does the heavy lifting between audits. Checking email plus phone against existing records before a new record is created is the single highest-leverage measure where form-driven intake is the main source of duplicates. One SaaS team reduced duplicate candidate records by 70% after pairing quarterly dedup audits with email as the unique identifier - the audit finds them, the intake check stops new ones.

Two more things keep the work current. First, retention windows shift and frameworks get republished, so re-confirm your jurisdiction windows with your DPO on the same quarterly cadence as the audit rather than trusting a static table. Second, the people who own this function are scarce - in Refolk's index, Recruiting or Talent Operations headcount in Germany is 6, led by KPMG Deutschland, and in the UK it is 3. If you are that person, the cadence above is what lets the database stay clean without your constant attention. Build the control once, and let it run.