Duplicate data appears when the same customer, product or order is entered in more than one place with no shared ID linking the entries. The fix has three parts: clean up the existing records carefully, set one rule for what counts as the same customer, and check every new entry against that rule before it gets saved.
This guide explains how duplicates form, what they cost in daily work, how to merge them without losing history, and includes a matching rules table you can use in any CRM or spreadsheet.
Why duplicate data happens
Very few duplicates come from carelessness. They come from having several ways to enter the same customer, and nothing that checks one against the other.
The usual causes are easy to spot once you look:
- Several lead sources. The same buyer enquires on IndiaMART, then fills in your website form with a different email.
- Different formats. 9876543210, +91 98765 43210 and 098765-43210 are one number, but software treats them as three.
- Name variations. "Sharma Traders", "Sharma Traders Pvt Ltd" and "M/s Sharma Traders" may all be one business.
- Disconnected teams. Sales uses a CRM, accounts uses Tally and neither can see the other's records.
- Imports. A spreadsheet uploaded twice can double a list in seconds.
- Automations with no check. A workflow that creates a contact on every enquiry will create a duplicate on every repeat enquiry.
What duplicates cost you
The cost shows up in small, repeated problems. Two salespeople call the same lead without knowing it.
A customer who has paid gets a payment reminder because the payment is on the other record. A complaint sits on one profile while sales pitches an upgrade from the other.
Your numbers also stop being reliable. If one customer is counted three times, your customer count goes up while your revenue per customer goes down, and your campaign lists send the same person the same message three times. On WhatsApp, repeat messages raise your costs and irritate the customer.
To size the problem, export your contacts, standardise the phone numbers and count how many repeat. That gives you a real percentage for your own data in under an hour, which is more useful than any industry average.
Duplicate matching rules table
Before merging anything, agree what counts as a match. Use tiers, so that software handles the clear cases and a person checks the rest. Adjust this table to suit your business.
| Tier | Match on | Action | Watch out for |
|---|---|---|---|
| Strong | Same system ID or customer code | Merge automatically | Old imports that reused codes |
| Strong | Same GSTIN | Merge company records after a quick review | One PAN can have a different GSTIN in each state |
| Medium | Same phone in standard format | Queue for review | Shared office lines and family phones |
| Medium | Same email address | Queue for review | Shared inboxes such as info@ or accounts@ |
| Weak | Similar company name and same city | Person checks | Branches and related businesses |
| Weak | Same person's name only | Do not merge | Common names |
Phone number deserves care in India, where a small business owner's mobile often doubles as the company number. One phone can correctly belong to two businesses run by one family. That is why it sits in the review tier.
How to clean up duplicates safely
1. Export a backup first
Treat every merge as permanent. HubSpot's duplicate guide says merged records cannot be reverted.
Zoho CRM's merge guide says the merged-in records are deleted permanently. Export the full module, with record IDs, before you start.
2. Standardise the formats
Convert phone numbers to one format, such as +919876543210. Put emails in lowercase, trim spaces, and strip suffixes like "Pvt Ltd" from a copy of the company name that you use only for matching. Many false differences disappear at this stage.
3. Decide which record survives
Pick a master record rule before you begin. The oldest record usually has the most history, while the most recently updated one usually has the best contact details. A good merge keeps the correct value for each field, along with all the activity.
4. Test on 50 records
Merge a small batch, then check that the notes, deals, invoices and open tasks all ended up on the surviving record. Ask the salesperson who owns those accounts whether anything looks wrong.
5. Merge in batches
Do the strong tier first and the review queue after it. Tell the team when each batch will run, so nobody edits those records during the merge.
What your CRM can do on its own
Check the built-in tools before you build anything. They differ more than you might expect.
| Feature | Zoho CRM | HubSpot |
|---|---|---|
| Prevent duplicates | Mark a field as unique and it alerts on a match | Surfaces likely duplicates as records are created |
| Find duplicates | De-duplicate tool matches on up to three fields you choose | Compares properties such as name, email, phone and company domain |
| Merge limit | Up to three records at a time | Pairs, individually or in bulk on eligible plans |
| Who can use it | The de-duplicate tool needs the top role or an Administrator profile | Professional or Enterprise; bulk needs Data Hub |
In Zoho, the de-duplicate tool merges exact copies automatically and asks you to resolve records with conflicting values. The emailed link to resolve those conflicts expires after three days, so run it when someone has time to finish. In HubSpot, custom duplicate rules need Data Hub Professional or Enterprise.
How to stop new duplicates
Cleaning without prevention means repeating the job in six months. Fix the entry points:
- Search before create. Every automation should look for an existing record first, then update it if one is found.
- Standardise at entry. Format phone numbers before they reach the CRM.
- Mark unique fields. Use the CRM's own settings for customer code, phone or GSTIN where they apply.
- Restrict imports. Let one or two people run bulk uploads, with a matching check first.
- Give every customer one ID. Store the CRM record ID in your other systems so they can link back.
- Review weekly. Send a short list of possible duplicates to one owner.
A sync that passes one shared ID between tools removes most of the risk. That is what well-built CRM automation does at each entry point, from IndiaMART to your website form.
How to tell the clean-up worked
Record a few numbers before you start and check them again after a month. You should see the duplicate rate fall and stay down.
| Measure | How to calculate it | What you want to see |
|---|---|---|
| Duplicate rate | Repeat phone numbers ÷ total contacts | Falls after the merge, then stays flat |
| New duplicates per week | Count from your weekly review | Close to zero |
| Entry point | Source field on each new duplicate | Shows which form or import to fix |
| Double follow-ups | Ask sales for examples each month | Fewer complaints from customers and staff |
Expect your total contact count to fall. Warn sales and marketing beforehand, so a smaller list does not look like lost data.
It also makes campaign reporting more accurate, because each person is counted once. Keep your backup file for a few months in case someone needs to check an old value.
Where AI helps, and where it does not
AI can compare messy text better than exact matching. It can tell that "Sharma Trdrs, Nashik" and "M/s Sharma Traders, Nasik Road" are probably the same business, and it can read names from forms in different spellings.
Use it to suggest matches, with a confidence level, and let a person approve anything below your threshold. An incorrect merge is worse than a duplicate, because it combines two customers' histories and may be impossible to undo.
Common questions
Should I fix the CRM or Tally first?
Start with the system your customer-facing team uses every day, which is usually the CRM. Ledgers in Tally need your accountant's involvement, since merging them affects financial records.
How often should I run a clean-up?
Do one full clean-up, then a weekly review of new possible duplicates. If that review keeps finding many, an entry point is still creating them, so find it and fix it there.
Can I delete duplicates in place of merging them?
Only when the record has no activity. Deleting can remove call notes, deals and tasks that a merge would have kept.
Start with an export
Export your contacts today, standardise the phone column and count the repeats. That single number tells you how large the job is and which entry point to fix first.
Duplicates are one reason a CRM stops being trusted. Stale phone numbers and missing updates are the other. Our CRM data guide looks at why records go out of date and which automations keep them current.
If you want a second pair of eyes on your matching rules, book a free 30-minute audit.
Want to know what you could automate?
Book a free 30-minute automation audit. We'll look at one process with you and tell you honestly whether automating it is worth it. See how we work or browse our services.
Book a free auditGet posts like this in your inbox once a week. Subscribe to NXT Weekly.
Keep reading
How to Automate Quotations in a Small Business, Step by Step
A seven-step setup for sending accurate quotes within the hour, routing big discounts to the owner and following up until the buyer answers, with a checklist and INR costs.
Tally vs Zoho Books: Which Is Better for Automating a Growing Business?
A side-by-side look at TallyPrime and Zoho Books for Indian businesses that want to automate invoices, approvals and reporting, with 2026 prices in rupees.