MarkLogic - detecting similar/duplicate names

I have a number of documents from different sources. Many of them reference a company name, but may have stored the information slightly differently. The name is a field in the documents.

I’d like to be able to detect variations on the same name, something like:

Ajax Company Incorporated
Ajax Co. Inc.
Ajax Company Inc.
Ajax Company
Ajax Company (formerly Ajax Unlimited)
etc

Does MarkLogic have any facility to query documents that have “similar” name as above? I’m not sure if there’s a more technical term that I should be searching for. Preferably for either the node client API or server-side js.

Posted by Roy Tang under notes at 14 Jul 2017 2:27am #marklogic #questions #stackoverflow #software development
Also on: stackexchange / 1

Comments

grtjn

2017-07-14 06:20:51

There are several options you could try, or combine:

Use thesaurus expansion to expand a search for one of these terms to any of the others. You can use semantics for that where you use owl:sameAs triples, or you could make use of the MarkLogic thsr library.
Normalize your data at ingest with a reverse lookup in the thesaurus or ontology of above. You could potentially tag found matches, and add the normalized name as an attribute for searches on the normalized term. You would normalize the search terms in the same manner.
Use spell:double-metaphone on each token in the name at ingest, and also on the search terms to search with those instead of the real name.

Search term expansion sounds like most straight-forward in this case, particularly since you are talking about mere spelling differences of terms like ‘Company’ and ‘Incorporated’.

HTH!

If you enjoyed my content for some reason, I'd love to hear from you! Here are some options:

You can buy me a coffee!
You can write a reply on your own site and submit the URL as a webmention via the form below.
Or you can just contact me!