# ANN : elasticsearch-entity-resolution plugin 0.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2013, 4:55pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598 "2013-09-13T16:55:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 13, 2013, 4:55pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/1 "2013-09-13T16:55:02Z")
Hi folks, I'm happy to announce my first release of Elasticsearch entity resolution plugin. > **[YannBrrd/elasticsearch-entity-resolution](https://github.com/YannBrrd/elasticsearch-entity-resolution/)** > > elasticsearch-entity-resolution - Elasticsearch entity resolution plugin based on Duke Hope you'll enjoy it ! Regards, Yann -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [September 15, 2013, 2:49pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/2 "2013-09-15T14:49:03Z")
Intriguing! Will play around with this, thanks for making it available! So the score of the each document is proportional to how "similar" the two items are, based on the deduplication parameters provided? Very naive question, I don't know anything about deduplication: how does this differ from doing regular searches that find similar documents (like a fuzzy or more-like-this query)? -Zach On Friday, September 13, 2013 12:55:02 PM UTC-4, Yann Barraud wrote: > Hi folks, > > I'm happy to announce my first release of Elasticsearch entity resolution > plugin. > > [GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity resolution plugin based on Duke](https://github.com/YannBrrd/elasticsearch-entity-resolution/) > > Hope you'll enjoy it ! > > Regards, > Yann -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 16, 2013, 7:06am UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/3 "2013-09-16T07:06:57Z")
Well, this differentiates search & scoring. Basically, it is not only about search matching items, it about know how similar an item is to another. This is really different. You might have a look there [ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) Quote : Can I filter by score? Not safely. You can always pick an arbitrary score value and then check the Hits object to see how many results have a score higher than that value (a Binary search might come in handy) but it really doesn't give you any meaningful information because of the way score is calculated... Scores As Percentages People frequently want to compute a "Percentage" from Lucene scores to determine what is a "100% perfect" match vs a "50%" match. This is also somethings called a "normalized score" Don't do this. Seriously. Stop trying to think about your problem this way, it's not going to end well. This project just fills in the gaps... Cordialement, Yann Barraud 2013/9/15 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > Intriguing! Will play around with this, thanks for making it available! > > So the score of the each document is proportional to how "similar" the two > items are, based on the deduplication parameters provided? Very naive > question, I don't know anything about deduplication: how does this differ > from doing regular searches that find similar documents (like a fuzzy or > more-like-this query)? > > -Zach > > On Friday, September 13, 2013 12:55:02 PM UTC-4, Yann Barraud wrote: > > > Hi folks, > > > > I'm happy to announce my first release of Elasticsearch entity resolution > > plugin. > > > > [https://github.com/YannBrrd/\*\*elasticsearch-entity-\*\*resolution/](https://github.com/YannBrrd/ **elasticsearch-entity-** resolution/)[https://github.com/YannBrrd/elasticsearch-entity-resolution/](https://github.com/YannBrrd/elasticsearch-entity-resolution/) > > > > Hope you'll enjoy it ! > > > > Regards, > > Yann > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [September 16, 2013, 1:05pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/4 "2013-09-16T13:05:39Z")
Gotcha, that makes sense. Thanks for the links. Will definitely spend some time playing with this, I can see it being very useful. Cheers, -Zach On Mon, Sep 16, 2013 at 3:06 AM, Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com)wrote: > Well, this differentiates search & scoring. Basically, it is not only > about search matching items, it about know how similar an item is to > another. This is really different. > > You might have a look there > [ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) > > Quote : > > Can I filter by score? > Not safely. You can always pick an arbitrary score value and then check the Hits object to see how many results have a score higher than that value (a Binary search might come in handy) but it really doesn't give you any meaningful information because of the way score is calculated... > > Scores As Percentages > > People frequently want to compute a "Percentage" from Lucene scores to determine what is a "100% perfect" match vs a "50%" match. This is also somethings called a "normalized score" > > Don't do this. > > Seriously. Stop trying to think about your problem this way, it's not going to end well. > > This project just fills in the gaps... > > Cordialement, > Yann Barraud > > 2013/9/15 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > > > Intriguing! Will play around with this, thanks for making it available! > > > > So the score of the each document is proportional to how "similar" the > > two items are, based on the deduplication parameters provided? Very naive > > question, I don't know anything about deduplication: how does this differ > > from doing regular searches that find similar documents (like a fuzzy or > > more-like-this query)? > > > > -Zach > > > > On Friday, September 13, 2013 12:55:02 PM UTC-4, Yann Barraud wrote: > > > > > Hi folks, > > > > > > I'm happy to announce my first release of Elasticsearch entity > > > resolution plugin. > > > > > > [https://github.com/YannBrrd/\*\*elasticsearch-entity-\*\*resolution/](https://github.com/YannBrrd/ **elasticsearch-entity-** resolution/)[https://github.com/YannBrrd/elasticsearch-entity-resolution/](https://github.com/YannBrrd/elasticsearch-entity-resolution/) > > > > > > Hope you'll enjoy it ! > > > > > > Regards, > > > Yann > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 16, 2013, 2:18pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/5 "2013-09-16T14:18:04Z")
Good to hear that ! 🙂 As you're an ES dev, I have a question for you : I'm looking for any code example about how to read data for index while instanciating the plugin ? I'd love to be able to store conf also within the index itself (type : entity-conf, doc named in the JSON payload). Do you have any pointer ? Any doc ? Thanks ! Cordialement, Yann Barraud 2013/9/16 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > Gotcha, that makes sense. Thanks for the links. Will definitely spend > some time playing with this, I can see it being very useful. > > Cheers, > -Zach > > On Mon, Sep 16, 2013 at 3:06 AM, Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com)wrote: > > > Well, this differentiates search & scoring. Basically, it is not only > > about search matching items, it about know how similar an item is to > > another. This is really different. > > > > You might have a look there > > [ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) > > > > Quote : > > > > Can I filter by score? > > Not safely. You can always pick an arbitrary score value and then check the Hits object to see how many results have a score higher than that value (a Binary search might come in handy) but it really doesn't give you any meaningful information because of the way score is calculated... > > > > Scores As Percentages > > > > People frequently want to compute a "Percentage" from Lucene scores to determine what is a "100% perfect" match vs a "50%" match. This is also somethings called a "normalized score" > > > > Don't do this. > > > > Seriously. Stop trying to think about your problem this way, it's not going to end well. > > > > This project just fills in the gaps... > > > > Cordialement, > > Yann Barraud > > > > 2013/9/15 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > > > > > Intriguing! Will play around with this, thanks for making it available! > > > > > > So the score of the each document is proportional to how "similar" the > > > two items are, based on the deduplication parameters provided? Very naive > > > question, I don't know anything about deduplication: how does this differ > > > from doing regular searches that find similar documents (like a fuzzy or > > > more-like-this query)? > > > > > > -Zach > > > > > > On Friday, September 13, 2013 12:55:02 PM UTC-4, Yann Barraud wrote: > > > > > > > Hi folks, > > > > > > > > I'm happy to announce my first release of Elasticsearch entity > > > > resolution plugin. > > > > > > > > [https://github.com/YannBrrd/\*\*elasticsearch-entity-\*\*resolution/](https://github.com/YannBrrd/ **elasticsearch-entity-** resolution/)[https://github.com/YannBrrd/elasticsearch-entity-resolution/](https://github.com/YannBrrd/elasticsearch-entity-resolution/) > > > > > > > > Hope you'll enjoy it ! > > > > > > > > Regards, > > > > Yann > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 16, 2013, 2:42pm UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/6 "2013-09-16T14:42:50Z")
Self reply : [https://github.com/imotov/elasticsearch-native-script-example/blob/master/src/main/java/org/elasticsearch/examples/nativescript/script/LookupScript.javathis](https://github.com/imotov/elasticsearch-native-script-example/blob/master/src/main/java/org/elasticsearch/examples/nativescript/script/LookupScript.javathis) should do the trick 😉 Cordialement, Yann Barraud 2013/9/16 Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com) > Good to hear that ! 🙂 > > As you're an ES dev, I have a question for you : I'm looking for any code > example about how to read data for index while instanciating the plugin ? > I'd love to be able to store conf also within the index itself (type : > entity-conf, doc named in the JSON payload). Do you have any pointer ? Any > doc ? > > Thanks ! > > Cordialement, > Yann Barraud > > 2013/9/16 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > > > Gotcha, that makes sense. Thanks for the links. Will definitely spend > > some time playing with this, I can see it being very useful. > > > > Cheers, > > -Zach > > > > On Mon, Sep 16, 2013 at 3:06 AM, Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com)wrote: > > > > > Well, this differentiates search & scoring. Basically, it is not only > > > about search matching items, it about know how similar an item is to > > > another. This is really different. > > > > > > You might have a look there > > > [ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & > > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) > > > > > > Quote : > > > > > > Can I filter by score? > > > Not safely. You can always pick an arbitrary score value and then check the Hits object to see how many results have a score higher than that value (a Binary search might come in handy) but it really doesn't give you any meaningful information because of the way score is calculated... > > > > > > Scores As Percentages > > > > > > People frequently want to compute a "Percentage" from Lucene scores to determine what is a "100% perfect" match vs a "50%" match. This is also somethings called a "normalized score" > > > > > > Don't do this. > > > > > > Seriously. Stop trying to think about your problem this way, it's not going to end well. > > > > > > This project just fills in the gaps... > > > > > > Cordialement, > > > Yann Barraud > > > > > > 2013/9/15 Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) > > > > > > > Intriguing! Will play around with this, thanks for making it available! > > > > > > > > So the score of the each document is proportional to how "similar" the > > > > two items are, based on the deduplication parameters provided? Very naive > > > > question, I don't know anything about deduplication: how does this differ > > > > from doing regular searches that find similar documents (like a fuzzy or > > > > more-like-this query)? > > > > > > > > -Zach > > > > > > > > On Friday, September 13, 2013 12:55:02 PM UTC-4, Yann Barraud wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I'm happy to announce my first release of Elasticsearch entity > > > > > resolution plugin. > > > > > > > > > > [https://github.com/YannBrrd/\*\*elasticsearch-entity-\*\*resolution/](https://github.com/YannBrrd/ **elasticsearch-entity-** resolution/)[https://github.com/YannBrrd/elasticsearch-entity-resolution/](https://github.com/YannBrrd/elasticsearch-entity-resolution/) > > > > > > > > > > Hope you'll enjoy it ! > > > > > > > > > > Regards, > > > > > Yann > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/N5UDIv9\_aeA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/N5UDIv9_aeA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/ann-elasticsearch-entity-resolution-plugin-0-1/13598/7 "2017-07-06T02:16:18Z")