# Entity/Identity resolution **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2013, 7:58am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559 "2013-04-12T07:58:35Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 12, 2013, 7:58am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/1 "2013-04-12T07:58:35Z")
Hi, I'm currently working on search engines, data cleaning and so on these last days. The challenge I'm facing right now is explaining that a search engine ((ie. ElasticSearch [http://www.elasticsearch.org](http://www.elasticsearch.org)in this case) on its own can not be used for identity resolution. Lucene posts made things easier ( [http://wiki.apache.org/lucene-java/ScoresAsPercentages](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & [http://wiki.apache.org/lucene-java/LuceneFAQ#Can\_I\_filter\_by\_score.3F](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) ). [http://wiki.apache.org/lucene-java/LuceneFAQ#Can\_I\_filter\_by\_score.3F](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) I've been playing with Duke [http://code.google.com/p/duke](http://code.google.com/p/duke)project also, for batch data deduplication. It's been very powerful, and covering requirements for batch needs. Now I'm wondering if there is not an opportunity to merge at some point the two projets to get some fast live identity resolution service. I'd say : 1. duke delegates data analysis & indexing to ElasticSearch [http://www.elasticsearch.org](http://www.elasticsearch.org)(as they both rely on Lucene indexes) 2. duke [http://code.google.com/p/duke](http://code.google.com/p/duke)turns into an ES plugin to get records matching query with Bayesian probability as an output. What do you guys think about it ? Regards, Yann Barraud -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 12, 2013, 8:47am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/2 "2013-04-12T08:47:16Z")
+1 I developed JDBC/CSV/RDF plugins with the intent of growing them later into a data cleaning / data transformation platform. There are many good things in Duke. I am developing an ES based engine for transforming large amounts of bibliographic data. Bibliograhic data is organized in graphs so I threw them over the fence into ES, and by simple term queries I can build connected graphs, using ES JSON sources for grouping algorithms, which I like to see them evolved into JSON-LD. Also I developed domain-specific (german language based) add-ons and match keys for processing unlinked bibliographic data (for matching authority data, author names e.g., for academic papers). I haven't analyzed the code how Duke uses Lucene but I think it is a very cool idea to use Duke code to become an ES app. GIventhe presentation in [Deduplication | PPT](http://de.slideshare.net/larsga/deduplication) there could be much improvement, for speed and versatility. For being a generic instrument, I am not sure if a single ES plugin can cover all the needs of the users out there, I think something like a simple dedup REST API hiding a river and a transformation step could be a start, to help (non-Java) developers setting up ES based dedup processing very fast. By adding more specific dedup methods, maybe several plugins could be feasible. Jörg Am 12.04.13 09:58, schrieb Yann Barraud: > Hi, > > I'm currently working on search engines, data cleaning and so on these > last days. The challenge I'm facing right now is explaining that a > search engine ((ie. Elasticsearch [http://www.elasticsearch.org](http://www.elasticsearch.org)in > this case) on its own can not be used for identity resolution. Lucene > posts made things easier > ([ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) ). > > I've been playing with Duke [http://code.google.com/p/duke](http://code.google.com/p/duke)project > also, for batch data deduplication. It's been very powerful, and > covering requirements for batch needs. > > Now I'm wondering if there is not an opportunity to merge at some > point the two projets to get some fast live identity resolution service. > > I'd say : > > 1. duke delegates data analysis & indexing to > Elasticsearch[http://www.elasticsearch.org](http://www.elasticsearch.org)(as they both rely on > Lucene indexes) > 2. duke [http://code.google.com/p/duke](http://code.google.com/p/duke)turns into an ES plugin to get > records matching query with Bayesian probability as an output. > > What do you guys think about it ? > > ## Regards, Yann Barraud > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 12, 2013, 9:06am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/3 "2013-04-12T09:06:58Z")
Hi Lars, I fact my sugesstion is a cross post to ES group. I think having a plugin would couver a simple service. Let's say I indexed all my data within ES, with the right configuration (analyzers, filters & co). The plugin could hook the query and add a Bayesian score for each and every match ES returns. In fact, this would imply using the class/methods you use to calculate the score on the records set ES returns... Le vendredi 12 avril 2013 10:24:46 UTC+2, Lars Marius Garshol a écrit : > - Yann Barraud > > > Now I'm wondering if there is not an opportunity to merge at some point > > the two projets to get some fast live identity resolution service. > > The Elasticsearch people will have to respond to what they think about > this. (I'm the maintainer of Duke, and have no real involvement with > Elasticsearch.) > > However, it is actually possible to do live identity resolution with Duke > already. The original project that I developed Duke for uses Duke that way, > to maintain a database of links between duplicate records that's updated as > the source data is updated. > > Basically, all you need for this is to implement a data source that > returns new records every time it's called. It's probably fairly easy to > extend the JDBC data source to handle that. (We use a different approach in > the original project.) > > > I'd say : > > • duke delegates data analysis & indexing to Elasticsearch (as they both > > rely on Lucene indexes) > > I guess the goal here is to exploit Elasticsearch's ability to > incrementally retrieve only new data, and that ES already has indexed up > the data. > > It is possible to fit this into Duke, basically by plugging in a new data > source and a new Database implementation. You can reuse bits of the > LuceneDatabase that's already there. > > > • duke turns into an ES plugin to get records matching query with > > Bayesian probability as an output. > > I'm not sure that's the right architecture, but it's possible to > implement. The Processor.match(Record, boolean) method essentially does > this already. > > -- > Lars Marius Garshol | Consultant > Bouvet ASA Sandakerveien 24C D11 Postboks 4430 Nydalen NO-0403 Oslo > Phone: +47 23 40 60 00 | Fax: +47 23 40 60 01 | Mobile: +47 98 21 55 50 > [http://www.bouvet.no](http://www.bouvet.no) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 12, 2013, 9:13am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/4 "2013-04-12T09:13:53Z")
Hi Jörg, The need I have at the moment is to get a simple service that could match an identity within my data and get a probability match over it. As far as I understand how Duke works, it first matches records with Lucene, then calculates Bayesian probability based on the XML configuration. If I'm right, what I would implement is : - A new Query type (ES plugin ?) involving Bayesian probability for each matching record, based an a specific type that would be the actual XML configuration duke uses. This would be an answer to Lucene posts I mentioned sooner. Lars has done a very good job with Duke. I think it is a very valuable asset, and I regret not having time to get my coding skills back at the moment, to help more. But if we manage, one way or another, to have both projects working together, I think we cold start building an incredibly fast entity resolution service. Le vendredi 12 avril 2013 10:47:16 UTC+2, Jörg Prante a écrit : > +1 > > I developed JDBC/CSV/RDF plugins with the intent of growing them later > into a data cleaning / data transformation platform. There are many good > things in Duke. I am developing an ES based engine for transforming > large amounts of bibliographic data. Bibliograhic data is organized in > graphs so I threw them over the fence into ES, and by simple term > queries I can build connected graphs, using ES JSON sources for grouping > algorithms, which I like to see them evolved into JSON-LD. Also I > developed domain-specific (german language based) add-ons and match keys > for processing unlinked bibliographic data (for matching authority data, > author names e.g., for academic papers). > > I haven't analyzed the code how Duke uses Lucene but I think it is a > very cool idea to use Duke code to become an ES app. GIventhe > presentation in [Deduplication | PPT](http://de.slideshare.net/larsga/deduplication) there > could be much improvement, for speed and versatility. > > For being a generic instrument, I am not sure if a single ES plugin can > cover all the needs of the users out there, I think something like a > simple dedup REST API hiding a river and a transformation step could be > a start, to help (non-Java) developers setting up ES based dedup > processing very fast. By adding more specific dedup methods, maybe > several plugins could be feasible. > > Jörg > > Am 12.04.13 09:58, schrieb Yann Barraud: > > > Hi, > > > > I'm currently working on search engines, data cleaning and so on these > > last days. The challenge I'm facing right now is explaining that a > > search engine ((ie. Elasticsearch [http://www.elasticsearch.org](http://www.elasticsearch.org)in > > this case) on its own can not be used for identity resolution. Lucene > > posts made things easier > > ([ScoresAsPercentages - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/ScoresAsPercentages) & > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F)). > > > > I've been playing with Duke [http://code.google.com/p/duke](http://code.google.com/p/duke)project > > also, for batch data deduplication. It's been very powerful, and > > covering requirements for batch needs. > > > > Now I'm wondering if there is not an opportunity to merge at some > > point the two projets to get some fast live identity resolution service. > > > > I'd say : > > > > 1. duke delegates data analysis & indexing to > > Elasticsearch[http://www.elasticsearch.org](http://www.elasticsearch.org)(as they both rely on > > Lucene indexes) > > 2. duke [http://code.google.com/p/duke](http://code.google.com/p/duke)turns into an ES plugin to get > > records matching query with Bayesian probability as an output. > > > > What do you guys think about it ? > > > > ## Regards, Yann Barraud > > > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 12, 2013, 9:28am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/5 "2013-04-12T09:28:48Z")
FYI there is activity in the area of text classification (which is a generalization of dedup) in the Lucene community, see > **[Text categorization with Lucene and Solr](https://de.slideshare.net/teofili/text-categorization-with-lucene-and-solr)** > > Text categorization with Lucene and Solr - Download as a PDF or view online for free and a simple bayesian classifier has been added recently to the Lucene codebase [http://lucene.apache.org/core/4\_2\_0/classification/org/apache/lucene/classification/SimpleNaiveBayesClassifier.html](http://lucene.apache.org/core/4_2_0/classification/org/apache/lucene/classification/SimpleNaiveBayesClassifier.html) I think this can be exposed by ES API as well. Jörg Am 12.04.13 11:13, schrieb Yann Barraud: > Hi Jörg, > > The need I have at the moment is to get a simple service that could > match an identity within my data and get a probability match over it. > > As far as I understand how Duke works, it first matches records with > Lucene, then calculates Bayesian probability based on the XML > configuration. > > If I'm right, what I would implement is : > > - A new Query type (ES plugin ?) involving Bayesian probability for > each matching record, based an a specific type that would be the > actual XML configuration duke uses. > > This would be an answer to Lucene posts I mentioned sooner. > > Lars has done a very good job with Duke. I think it is a very valuable > asset, and I regret not having time to get my coding skills back at > the moment, to help more. But if we manage, one way or another, to > have both projects working together, I think we cold start building an > incredibly fast entity resolution service. > > Le vendredi 12 avril 2013 10:47:16 UTC+2, Jörg Prante a écrit : > > ``` > +1 > > I developed JDBC/CSV/RDF plugins with the intent of growing them > later > into a data cleaning / data transformation platform. There are > many good > things in Duke. I am developing an ES based engine for transforming > large amounts of bibliographic data. Bibliograhic data is > organized in > graphs so I threw them over the fence into ES, and by simple term > queries I can build connected graphs, using ES JSON sources for > grouping > algorithms, which I like to see them evolved into JSON-LD. Also I > developed domain-specific (german language based) add-ons and > match keys > for processing unlinked bibliographic data (for matching authority > data, > author names e.g., for academic papers). > > I haven't analyzed the code how Duke uses Lucene but I think it is a > very cool idea to use Duke code to become an ES app. GIventhe > presentation in http://de.slideshare.net/larsga/deduplication > there > could be much improvement, for speed and versatility. > > For being a generic instrument, I am not sure if a single ES > plugin can > cover all the needs of the users out there, I think something like a > simple dedup REST API hiding a river and a transformation step > could be > a start, to help (non-Java) developers setting up ES based dedup > processing very fast. By adding more specific dedup methods, maybe > several plugins could be feasible. > > Jörg > > Am 12.04.13 09:58, schrieb Yann Barraud: > > Hi, > > > > I'm currently working on search engines, data cleaning and so on > these > > last days. The challenge I'm facing right now is explaining that a > > search engine ((ie. ElasticSearch in > > this case) on its own can not be used for identity resolution. > Lucene > > posts made things easier > > (http://wiki.apache.org/lucene-java/ScoresAsPercentages > & > > > http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F > > ). > > > > I've been playing with Duke project > > also, for batch data deduplication. It's been very powerful, and > > covering requirements for batch needs. > > > > Now I'm wondering if there is not an opportunity to merge at some > > point the two projets to get some fast live identity resolution > service. > > > > I'd say : > > > > 1. duke delegates data analysis & indexing to > > ElasticSearch >(as they both rely on > > Lucene indexes) > > 2. duke turns into an ES plugin > to get > > records matching query with Bayesian probability as an output. > > > > What do you guys think about it ? > > > > Regards, > > Yann Barraud > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from > it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out > . > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 12, 2013, 9:34am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/6 "2013-04-12T09:34:02Z")
Thanks ! I'll look at it ASAP ! Le vendredi 12 avril 2013 11:28:48 UTC+2, Jörg Prante a écrit : > FYI there is activity in the area of text classification (which is a > generalization of dedup) in the Lucene community, see > > [Text categorization with Lucene and Solr | PPT](http://de.slideshare.net/teofili/text-categorization-with-lucene-and-solr) > > and a simple bayesian classifier has been added recently to the Lucene > codebase > > [SimpleNaiveBayesClassifier (Lucene 4.2.0 API)](http://lucene.apache.org/core/4_2_0/classification/org/apache/lucene/classification/SimpleNaiveBayesClassifier.html) > > I think this can be exposed by ES API as well. > > Jörg > > Am 12.04.13 11:13, schrieb Yann Barraud: > > > Hi Jörg, > > > > The need I have at the moment is to get a simple service that could > > match an identity within my data and get a probability match over it. > > > > As far as I understand how Duke works, it first matches records with > > Lucene, then calculates Bayesian probability based on the XML > > configuration. > > > > If I'm right, what I would implement is : > > > > - A new Query type (ES plugin ?) involving Bayesian probability for > > each matching record, based an a specific type that would be the > > actual XML configuration duke uses. > > > > This would be an answer to Lucene posts I mentioned sooner. > > > > Lars has done a very good job with Duke. I think it is a very valuable > > asset, and I regret not having time to get my coding skills back at > > the moment, to help more. But if we manage, one way or another, to > > have both projects working together, I think we cold start building an > > incredibly fast entity resolution service. > > > > Le vendredi 12 avril 2013 10:47:16 UTC+2, Jörg Prante a écrit : > > > > ``` > > +1 > > > > I developed JDBC/CSV/RDF plugins with the intent of growing them > > later > > into a data cleaning / data transformation platform. There are > > many good > > things in Duke. I am developing an ES based engine for transforming > > large amounts of bibliographic data. Bibliograhic data is > > organized in > > graphs so I threw them over the fence into ES, and by simple term > > queries I can build connected graphs, using ES JSON sources for > > grouping > > algorithms, which I like to see them evolved into JSON-LD. Also I > > developed domain-specific (german language based) add-ons and > > match keys > > for processing unlinked bibliographic data (for matching authority > > data, > > author names e.g., for academic papers). > > > > I haven't analyzed the code how Duke uses Lucene but I think it is a > > very cool idea to use Duke code to become an ES app. GIventhe > > presentation in http://de.slideshare.net/larsga/deduplication > > there > > could be much improvement, for speed and versatility. > > > > For being a generic instrument, I am not sure if a single ES > > plugin can > > cover all the needs of the users out there, I think something like a > > simple dedup REST API hiding a river and a transformation step > > could be > > a start, to help (non-Java) developers setting up ES based dedup > > processing very fast. By adding more specific dedup methods, maybe > > several plugins could be feasible. > > > > Jörg > > > > Am 12.04.13 09:58, schrieb Yann Barraud: > > > Hi, > > > > > > I'm currently working on search engines, data cleaning and so on > > these > > > last days. The challenge I'm facing right now is explaining that a > > > search engine ((ie. ElasticSearch in > > > > ``` > > > ``` > > > this case) on its own can not be used for identity resolution. > > Lucene > > > posts made things easier > > > (http://wiki.apache.org/lucene-java/ScoresAsPercentages > > & > > > > > > > ``` > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) > > > ``` > > < > > > > ``` > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F)\> > > > ``` > > ). > > > > > > I've been playing with Duke project > > > > ``` > > > ``` > > > also, for batch data deduplication. It's been very powerful, and > > > covering requirements for batch needs. > > > > > > Now I'm wondering if there is not an opportunity to merge at some > > > point the two projets to get some fast live identity resolution > > service. > > > > > > I'd say : > > > > > > 1. duke delegates data analysis & indexing to > > > ElasticSearch > >(as they both rely on > > > Lucene indexes) > > > 2. duke turns into an ES plugin > > to get > > > records matching query with Bayesian probability as an output. > > > > > > What do you guys think about it ? > > > > > > Regards, > > > Yann Barraud > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from > > it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit https://groups.google.com/groups/opt_out > > . > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [April 12, 2013, 10:43am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/7 "2013-04-12T10:43:38Z")
On Fri, 2013-04-12 at 11:28 +0200, Jörg Prante wrote: > FYI there is activity in the area of text classification (which is a > generalization of dedup) in the Lucene community, see > > [Text categorization with Lucene and Solr | PPT](http://de.slideshare.net/teofili/text-categorization-with-lucene-and-solr) > > and a simple bayesian classifier has been added recently to the Lucene > codebase > > [SimpleNaiveBayesClassifier (Lucene 4.2.0 API)](http://lucene.apache.org/core/4_2_0/classification/org/apache/lucene/classification/SimpleNaiveBayesClassifier.html) > > I think this can be exposed by ES API as well. Great presentations - looks exciting. > Jörg > > Am 12.04.13 11:13, schrieb Yann Barraud: > > > Hi Jörg, > > > > The need I have at the moment is to get a simple service that could > > match an identity within my data and get a probability match over it. > > > > As far as I understand how Duke works, it first matches records with > > Lucene, then calculates Bayesian probability based on the XML > > configuration. > > > > If I'm right, what I would implement is : > > > > - A new Query type (ES plugin ?) involving Bayesian probability for > > each matching record, based an a specific type that would be the > > actual XML configuration duke uses. > > > > This would be an answer to Lucene posts I mentioned sooner. > > > > Lars has done a very good job with Duke. I think it is a very valuable > > asset, and I regret not having time to get my coding skills back at > > the moment, to help more. But if we manage, one way or another, to > > have both projects working together, I think we cold start building an > > incredibly fast entity resolution service. > > > > Le vendredi 12 avril 2013 10:47:16 UTC+2, Jörg Prante a écrit : > > > > ``` > > +1 > > > > I developed JDBC/CSV/RDF plugins with the intent of growing them > > later > > into a data cleaning / data transformation platform. There are > > many good > > things in Duke. I am developing an ES based engine for transforming > > large amounts of bibliographic data. Bibliograhic data is > > organized in > > graphs so I threw them over the fence into ES, and by simple term > > queries I can build connected graphs, using ES JSON sources for > > grouping > > algorithms, which I like to see them evolved into JSON-LD. Also I > > developed domain-specific (german language based) add-ons and > > match keys > > for processing unlinked bibliographic data (for matching authority > > data, > > author names e.g., for academic papers). > > > > I haven't analyzed the code how Duke uses Lucene but I think it is a > > very cool idea to use Duke code to become an ES app. GIventhe > > presentation in http://de.slideshare.net/larsga/deduplication > > there > > could be much improvement, for speed and versatility. > > > > For being a generic instrument, I am not sure if a single ES > > plugin can > > cover all the needs of the users out there, I think something like a > > simple dedup REST API hiding a river and a transformation step > > could be > > a start, to help (non-Java) developers setting up ES based dedup > > processing very fast. By adding more specific dedup methods, maybe > > several plugins could be feasible. > > > > Jörg > > > > Am 12.04.13 09:58, schrieb Yann Barraud: > > > Hi, > > > > > > I'm currently working on search engines, data cleaning and so on > > these > > > last days. The challenge I'm facing right now is explaining that a > > > search engine ((ie. ElasticSearch in > > > this case) on its own can not be used for identity resolution. > > Lucene > > > posts made things easier > > > (http://wiki.apache.org/lucene-java/ScoresAsPercentages > > & > > > > > http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F > > > > ). > > > > > > I've been playing with Duke project > > > also, for batch data deduplication. It's been very powerful, and > > > covering requirements for batch needs. > > > > > > Now I'm wondering if there is not an opportunity to merge at some > > > point the two projets to get some fast live identity resolution > > service. > > > > > > I'd say : > > > > > > 1. duke delegates data analysis & indexing to > > > ElasticSearch > >(as they both rely on > > > Lucene indexes) > > > 2. duke turns into an ES plugin > > to get > > > records matching query with Bayesian probability as an output. > > > > > > What do you guys think about it ? > > > > > > Regards, > > > Yann Barraud > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from > > it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit https://groups.google.com/groups/opt_out > > . > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 12, 2013, 11:06am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/8 "2013-04-12T11:06:38Z")
Indeed ! I'm currently browsing Duke code. It is quite simple & clear. I wish I could scrapbook a plugin, but do not know yet where to start. Any pointer you could give me ? Le vendredi 12 avril 2013 12:43:38 UTC+2, Clinton Gormley a écrit : > On Fri, 2013-04-12 at 11:28 +0200, Jörg Prante wrote: > > > FYI there is activity in the area of text classification (which is a > > generalization of dedup) in the Lucene community, see > > [Text categorization with Lucene and Solr | PPT](http://de.slideshare.net/teofili/text-categorization-with-lucene-and-solr) > > > and a simple bayesian classifier has been added recently to the Lucene > > codebase > > [SimpleNaiveBayesClassifier (Lucene 4.2.0 API)](http://lucene.apache.org/core/4_2_0/classification/org/apache/lucene/classification/SimpleNaiveBayesClassifier.html) > > > I think this can be exposed by ES API as well. > > Great presentations - looks exciting. > > > Jörg > > > > Am 12.04.13 11:13, schrieb Yann Barraud: > > > > > Hi Jörg, > > > > > > The need I have at the moment is to get a simple service that could > > > match an identity within my data and get a probability match over it. > > > > > > As far as I understand how Duke works, it first matches records with > > > Lucene, then calculates Bayesian probability based on the XML > > > configuration. > > > > > > If I'm right, what I would implement is : > > > > > > - A new Query type (ES plugin ?) involving Bayesian probability for > > > each matching record, based an a specific type that would be the > > > actual XML configuration duke uses. > > > > > > This would be an answer to Lucene posts I mentioned sooner. > > > > > > Lars has done a very good job with Duke. I think it is a very valuable > > > asset, and I regret not having time to get my coding skills back at > > > the moment, to help more. But if we manage, one way or another, to > > > have both projects working together, I think we cold start building an > > > incredibly fast entity resolution service. > > > > > > Le vendredi 12 avril 2013 10:47:16 UTC+2, Jörg Prante a écrit : > > > > > > ``` > > > +1 > > > > > > I developed JDBC/CSV/RDF plugins with the intent of growing them > > > later > > > into a data cleaning / data transformation platform. There are > > > many good > > > things in Duke. I am developing an ES based engine for > > > > > > ``` > > transforming > > > > ``` > > > large amounts of bibliographic data. Bibliograhic data is > > > organized in > > > graphs so I threw them over the fence into ES, and by simple term > > > queries I can build connected graphs, using ES JSON sources for > > > grouping > > > algorithms, which I like to see them evolved into JSON-LD. Also I > > > developed domain-specific (german language based) add-ons and > > > match keys > > > for processing unlinked bibliographic data (for matching authority > > > data, > > > author names e.g., for academic papers). > > > > > > I haven't analyzed the code how Duke uses Lucene but I think it is > > > > > > ``` > > a > > > > ``` > > > very cool idea to use Duke code to become an ES app. GIventhe > > > presentation in http://de.slideshare.net/larsga/deduplication > > > there > > > could be much improvement, for speed and versatility. > > > > > > For being a generic instrument, I am not sure if a single ES > > > plugin can > > > cover all the needs of the users out there, I think something like > > > > > > ``` > > a > > > > ``` > > > simple dedup REST API hiding a river and a transformation step > > > could be > > > a start, to help (non-Java) developers setting up ES based dedup > > > processing very fast. By adding more specific dedup methods, maybe > > > several plugins could be feasible. > > > > > > Jörg > > > > > > Am 12.04.13 09:58, schrieb Yann Barraud: > > > > Hi, > > > > > > > > I'm currently working on search engines, data cleaning and so on > > > these > > > > last days. The challenge I'm facing right now is explaining that > > > > > > ``` > > a > > > > ``` > > > > search engine ((ie. ElasticSearch in > > > > > > ``` > > > > ``` > > > > this case) on its own can not be used for identity resolution. > > > Lucene > > > > posts made things easier > > > > (http://wiki.apache.org/lucene-java/ScoresAsPercentages > > > & > > > > > > > > > > ``` > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F) > > > > ``` > > > < > > > > > > ``` > > [LuceneFAQ - Apache Lucene (Java) - Apache Software Foundation](http://wiki.apache.org/lucene-java/LuceneFAQ#Can_I_filter_by_score.3F)\> > > > > ``` > > > ). > > > > > > > > I've been playing with Duke project > > > > > > ``` > > > > ``` > > > > also, for batch data deduplication. It's been very powerful, and > > > > covering requirements for batch needs. > > > > > > > > Now I'm wondering if there is not an opportunity to merge at > > > > > > ``` > > some > > > > ``` > > > > point the two projets to get some fast live identity resolution > > > service. > > > > > > > > I'd say : > > > > > > > > 1. duke delegates data analysis & indexing to > > > > ElasticSearch > > >(as they both rely on > > > > Lucene indexes) > > > > 2. duke turns into an ES plugin > > > to get > > > > records matching query with Bayesian probability as an > > > > > > ``` > > output. > > > > ``` > > > > > > > > What do you guys think about it ? > > > > > > > > Regards, > > > > Yann Barraud > > > > -- > > > > You received this message because you are subscribed to the > > > > > > ``` > > Google > > > > ``` > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from > > > it, send > > > > an email to elasticsearc...@googlegroups.com. > > > > For more options, visit https://groups.google.com/groups/opt_out > > > . > > > > > > > > > > > > > > ``` > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 12, 2013, 1:22pm UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/9 "2013-04-12T13:22:33Z")
I'm not very familiar with the Duke code internals. The Lucene approach is something like this: first, throw a lot of training documents into a Lucene index. Then call "train" method in SimpleNaiveBayesClassfiier and give the classifier a name. In a second step, you apply the SimpleNaiveBayesClassfiier to a given docuemnt (ES JSON) with "getAssignedClass". The JSON is then enriched with a field where the found class is written in, and the probability is given in "getScore()" which could also be added to the given JSON. The JSON could be handled like in the current "MoreLikeThis" ES action. I would try to figure out if the Duke approach is more versatile than the simple Lucene approach, and if the Lucene approach could somehow be extended to fit the Duke methods. And, you might consider how a "classify API" in ES HTTP REST could look like... something like "\_train" and "\_assign" endpoints could be added... Jörg Am 12.04.13 13:06, schrieb Yann Barraud: > Indeed ! I'm currently browsing Duke code. It is quite simple & clear. > I wish I could scrapbook a plugin, but do not know yet where to start. > > Any pointer you could give me ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 14, 2013, 10:48am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/10 "2013-04-14T10:48:55Z")
Hi J¨rog, In fact what I have in mind (which might be erroneous) is the following : - Enrich (extend) a request, by adding some matching configuration, more or less the way Duke is configured : ```auto { "term" : { "user" : "kimchy" }, } ``` Le vendredi 12 avril 2013 15:22:33 UTC+2, Jörg Prante a écrit : > I'm not very familiar with the Duke code internals. > > The Lucene approach is something like this: first, throw a lot of > training documents into a Lucene index. Then call "train" method in > SimpleNaiveBayesClassfiier and give the classifier a name. In a second > step, you apply the SimpleNaiveBayesClassfiier to a given docuemnt (ES > JSON) with "getAssignedClass". The JSON is then enriched with a field > where the found class is written in, and the probability is given in > "getScore()" which could also be added to the given JSON. The JSON could > be handled like in the current "MoreLikeThis" ES action. > > I would try to figure out if the Duke approach is more versatile than > the simple Lucene approach, and if the Lucene approach could somehow be > extended to fit the Duke methods. > > And, you might consider how a "classify API" in ES HTTP REST could look > like... something like "\_train" and "\_assign" endpoints could be added... > > Jörg > > Am 12.04.13 13:06, schrieb Yann Barraud: > > > Indeed ! I'm currently browsing Duke code. It is quite simple & clear. > > I wish I could scrapbook a plugin, but do not know yet where to start. > > > > Any pointer you could give me ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 14, 2013, 11:00am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/11 "2013-04-14T11:00:30Z")
Hi Jörg, In fact what I have in mind (which might be erroneous) is the following : - Enrich (extend/create) a request, by adding some matching configuration, more or less the way Duke is configured[http://code.google.com/p/duke/wiki/XMLConfig](http://code.google.com/p/duke/wiki/XMLConfig): { "term" : { "user" : "kimchy" }, "entity\_resolution" : { "field" : { "name" : "user", "cleaner" : "asciifolding, toLowerCase", "comparator" : "levenshtein", "min" : "0.3", "max" : "0.8" } } Then, for each match, apply this configuration calling double no.priv.garshol.duke.Processor.compare(Record, Record) where the returned value is the probability the matching value is the same that the requested one. The answer should be the exact way it is now, more a field (called \_matching\_probability ?) containing this value. Do you guys think this is a good way to do this ? I would then implement it as a plugin. But I have to find out how to (and if I have to) extend or create new search request and then enrich QueryDSL. PS : spent a few hours awake this night thinking about it. I think this should be whether an option within Term Query or a new Query based on Term Query. Regards, Yann Le vendredi 12 avril 2013 15:22:33 UTC+2, Jörg Prante a écrit : I'm not very familiar with the Duke code internals. > The Lucene approach is something like this: first, throw a lot of > training documents into a Lucene index. Then call "train" method in > SimpleNaiveBayesClassfiier and give the classifier a name. In a second > step, you apply the SimpleNaiveBayesClassfiier to a given docuemnt (ES > JSON) with "getAssignedClass". The JSON is then enriched with a field > where the found class is written in, and the probability is given in > "getScore()" which could also be added to the given JSON. The JSON could > be handled like in the current "MoreLikeThis" ES action. > > I would try to figure out if the Duke approach is more versatile than > the simple Lucene approach, and if the Lucene approach could somehow be > extended to fit the Duke methods. > > And, you might consider how a "classify API" in ES HTTP REST could look > like... something like "\_train" and "\_assign" endpoints could be added... > > Jörg > > Am 12.04.13 13:06, schrieb Yann Barraud: > > > Indeed ! I'm currently browsing Duke code. It is quite simple & clear. > > I wish I could scrapbook a plugin, but do not know yet where to start. > > > > Any pointer you could give me ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 14, 2013, 4:26pm UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/12 "2013-04-14T16:26:29Z")
Yann, you should check out how FuzzyQuery works. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > **[Lucene's FuzzyQuery is 100 times faster in 4.0](https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html)** > > There are many exciting improvements in Lucene's eventual 4.0 (trunk) release, but the awesome speedup to FuzzyQuery really stands out, not... Jörg Am 14.04.13 13:00, schrieb Yann Barraud: > PS : spent a few hours awake this night thinking about it. I think > this should be whether an option within Term Query or a new Query > based on Term Query. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 15, 2013, 6:10pm UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/13 "2013-04-15T18:10:12Z")
Hi, I looked at the FuzzyQuery. It is really interesting for what I have in mind. But if I understood well, all queries are Licene Queries, and thus results are all parsed the same way. Am I right ? This means I can not inject more a treatment (Bayesain scoring) to my results even with a new Query plugin ? Cordialement, Yann Barraud 2013/4/14 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > Yann, you should check out how FuzzyQuery works. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **guide/reference/query-dsl/** fuzzy-query/)[http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/](http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/) > > [http://blog.mikemccandless](http://blog.mikemccandless). **com/2011/03/lucenes-** > fuzzyquery-is-100-times-\*\*faster.html[http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html](http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html) > > Jörg > > Am 14.04.13 13:00, schrieb Yann Barraud: > > PS : spent a few hours awake this night thinking about it. I think this > > > should be whether an option within Term Query or a new Query based on Term > > Query. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > topic/elasticsearch/\*\*YMNaGOxTda4/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [April 16, 2013, 8:20am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/14 "2013-04-16T08:20:32Z")
Ok, digging & digging, I found out implementig a CustomScoreQuery with a native script is a more standard answer regarding ES architecture. Going this way. Hope I won't get lost once more. 😉 Cordialement, Yann Barraud 2013/4/15 Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com) > Hi, > > I looked at the FuzzyQuery. It is really interesting for what I have in > mind. But if I understood well, all queries are Licene Queries, and thus > results are all parsed the same way. Am I right ? > > This means I can not inject more a treatment (Bayesain scoring) to my > results even with a new Query plugin ? > > Cordialement, > Yann Barraud > > 2013/4/14 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > > > Yann, you should check out how FuzzyQuery works. > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **guide/reference/query-dsl/** fuzzy-query/)[http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/](http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/) > > > > [http://blog.mikemccandless](http://blog.mikemccandless). **com/2011/03/lucenes-** > > fuzzyquery-is-100-times-\*\*faster.html[http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html](http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html) > > > > Jörg > > > > Am 14.04.13 13:00, schrieb Yann Barraud: > > > > PS : spent a few hours awake this night thinking about it. I think this > > > > > should be whether an option within Term Query or a new Query based on Term > > > Query. > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > > topic/elasticsearch/\*\*YMNaGOxTda4/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US) > > . > > To unsubscribe from this group and all its topics, send an email to > > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 12, 2013, 2:46pm UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/15 "2013-09-12T14:46:05Z")
Project finally on its way ! > **[GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity...](https://github.com/YannBrrd/elasticsearch-entity-resolution)** > > Elasticsearch entity resolution plugin based on Duke - GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity resolution plugin based on Duke Le mardi 16 avril 2013 10:20:32 UTC+2, Yann Barraud a écrit : > Ok, digging & digging, I found out implementig a CustomScoreQuery with a > native script is a more standard answer regarding ES architecture. > > Going this way. > > Hope I won't get lost once more. 😉 > > Cordialement, > Yann Barraud > > 2013/4/15 Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com) > > > Hi, > > > > I looked at the FuzzyQuery. It is really interesting for what I have in > > mind. But if I understood well, all queries are Licene Queries, and thus > > results are all parsed the same way. Am I right ? > > > > This means I can not inject more a treatment (Bayesain scoring) to my > > results even with a new Query plugin ? > > > > Cordialement, > > Yann Barraud > > > > 2013/4/14 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > > > > > Yann, you should check out how FuzzyQuery works. > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **guide/reference/query-dsl/** fuzzy-query/)[http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/](http://www.elasticsearch.org/guide/reference/query-dsl/fuzzy-query/) > > > > > > [http://blog.mikemccandless](http://blog.mikemccandless). **com/2011/03/lucenes-** > > > fuzzyquery-is-100-times-\*\*faster.html[http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html](http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html) > > > > > > Jörg > > > > > > Am 14.04.13 13:00, schrieb Yann Barraud: > > > > > > PS : spent a few hours awake this night thinking about it. I think this > > > > > > > should be whether an option within Term Query or a new Query based on Term > > > > Query. > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > > > topic/elasticsearch/\*\*YMNaGOxTda4/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/YMNaGOxTda4/unsubscribe?hl=en-US) > > > . > > > To unsubscribe from this group and all its topics, send an email to > > > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [September 12, 2013, 2:46pm UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/16 "2013-09-12T14:46:34Z")
Project finally on its way !! > **[GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity...](https://github.com/YannBrrd/elasticsearch-entity-resolution)** > > Elasticsearch entity resolution plugin based on Duke - GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity resolution plugin based on Duke Le vendredi 12 avril 2013 14:54:04 UTC+2, Lars Marius Garshol a écrit : > - Yann Barraud > > > I fact my sugesstion is a cross post to ES group. > > Yeah, I saw. > > > Let's say I indexed all my data within ES, with the right configuration > > (analyzers, filters & co). The plugin could hook the query and add a > > Bayesian score for each and every match ES returns. In fact, this would > > imply using the class/methods you use to calculate the score on the records > > set ES returns… > > Yes, that's definitely possible. Basically, you could do this by using > Processor.compare(Record, Record), which is already there and public. You > could use that without having any data sources at all. > > I think if you want to take this further, probably the best thing to do is > to simply write the plugin. It doesn't sound like it should be hard at all. > I'd be very happy to add it to Duke as a contribution. Or it could go into > Elasticsearch, if they want it. Either way is fine with me. > > -- > Lars Marius Garshol | Consultant > Bouvet ASA Sandakerveien 24C D11 Postboks 4430 Nydalen NO-0403 Oslo > Phone: +47 23 40 60 00 | Fax: +47 23 40 60 01 | Mobile: +47 98 21 55 50 > [http://www.bouvet.no](http://www.bouvet.no) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/entity-identity-resolution/11559/17 "2017-07-06T02:16:53Z")