# Probabilistic Record Linkage using Elastic Search

**URL:** <https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783>\
**Category:** Elasticsearch\
**Created:** [August 20, 2015, 5:47pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783 "2015-08-20T17:47:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![SAURAV\_PAUL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurav_paul/32/1277_2.png) [@SAURAV\_PAUL](https://discuss.elastic.co/u/SAURAV_PAUL)\
**Post date:** [August 20, 2015, 5:47pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/1 "2015-08-20T17:47:44Z")

</div>

Hi,

I would like to know How do I use Elastic Search to implement Probabilistic Record Linkage across multiple data set?  
Do we I need to pre-process the data from multiple source first then push the data into ES or Shall I load the data directly push the data into ES?

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [August 20, 2015, 6:04pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/2 "2015-08-20T18:04:51Z")

</div>

You can load the data sets and then you can use fuzzy matching and/or implement function score scripts. See for example [https://github.com/YannBrrd/elasticsearch-entity-resolution](https://github.com/YannBrrd/elasticsearch-entity-resolution)

---

<div class="post-metadata">

**Author:** ![SAURAV\_PAUL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurav_paul/32/1277_2.png) [@SAURAV\_PAUL](https://discuss.elastic.co/u/SAURAV_PAUL)\
**Post date:** [August 20, 2015, 8:10pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/3 "2015-08-20T20:10:52Z")

</div>

It doesn't support 1.7.1 which internally uses Duke. This supports till 1.4.1

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [August 20, 2015, 9:47pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/4 "2015-08-20T21:47:30Z")

</div>

Did you try it?

It compiles against Elasticsearch 1.7.1 without any changes, so it will work.

---

<div class="post-metadata">

**Author:** ![SAURAV\_PAUL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurav_paul/32/1277_2.png) [@SAURAV\_PAUL](https://discuss.elastic.co/u/SAURAV_PAUL)\
**Post date:** [August 21, 2015, 3:57pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/5 "2015-08-21T15:57:39Z")

</div>

Yup I tried it compiles against 1.7.1 only if we skip the test cases.

---

<div class="post-metadata">

**Author:** ![SAURAV\_PAUL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurav_paul/32/1277_2.png) [@SAURAV\_PAUL](https://discuss.elastic.co/u/SAURAV_PAUL)\
**Post date:** [August 21, 2015, 7:02pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/6 "2015-08-21T19:02:48Z")

</div>

I took the source code and tried to build/compile it just upgrading the ES version. If I use mvn clean install -DskipTests=true it compiles the code and generates the jar however if I use it only mvn clean install it fails it throws up exception NoClassDef from test class

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [August 24, 2015, 7:09am UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/7 "2015-08-24T07:09:23Z")

</div>

Test must not fail. You have to update to Lucene 4.10.4 for ES 1.7.1. Edit pom.xml to set versions:

```auto
 <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <elasticsearch.version>1.7.1</elasticsearch.version>
        <lucene.version>4.10.4</lucene.version>
        <duke.version>1.2</duke.version>
    </properties>

```

---

<div class="post-metadata">

**Author:** ![SAURAV\_PAUL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saurav_paul/32/1277_2.png) [@SAURAV\_PAUL](https://discuss.elastic.co/u/SAURAV_PAUL)\
**Post date:** [August 24, 2015, 5:35pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/8 "2015-08-24T17:35:06Z")

</div>

Thank you. Already did that. it works now. Now the question is, How should I approach the problem statement in ES? I mean I have two datasets I have to perform the record Linking across dataset. Do I need to index both of the dataset? I am confused with how does this entity-resolution plug-in works with multiple datasets?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:54pm UTC](https://discuss.elastic.co/t/probabilistic-record-linkage-using-elastic-search/27783/9 "2017-07-05T23:54:07Z")

</div>