# Indexing a large Nx N matrix of similarity with ES **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2014, 10:09pm UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210 "2014-04-25T22:09:13Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![nicom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicom/32/488_2.png) [@nicom](https://discuss.elastic.co/u/nicom)\ **Post date:** [April 25, 2014, 10:09pm UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210/1 "2014-04-25T22:09:13Z")
I have N documents containing attributes. I needed to precompute a special similarity measure between each pairwise of documents. Now I would to understand how to index and search using ES to answer a query like "Retrieve me the Top N documents that are the most similar to document ID 1 and having as fieldA = 1" and facets the results according to a given field -- I was thinking to create an index of documnts with all the associated pairwises as attributes,like: Doc id: 1 field1: 7 field2: 10 sim\_doc\_id2: 10 sim\_doc\_id3: 8 sim\_doc\_id4: 12 ... sim\_doc\_idN: 12 Doc id: 2 field1: 5 field2: 2 sim\_doc\_id1: 10 sim\_doc\_id3: 3 sim\_doc\_id4: 2 ... sim\_doc\_idN: 10 .. issue with such design The number of generated fields per document is very large for me (10K) and I am not sure how to search efficiently (I tried a script score like return doc['sim\_doc\_id1'] + field1) but it was quite slow.. especially compared to a stupid loop in Java. however I would to use the aggregation framework of ES to create facets of the results. Do you have any recommendation / guideline to handle this pb? Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Rob\_Styles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rob_styles/32/1612_2.png) [@Rob\_Styles](https://discuss.elastic.co/u/Rob_Styles)\ **Post date:** [April 25, 2014, 10:25pm UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210/2 "2014-04-25T22:25:23Z")
Not elasticsearch, but I'm doing a similar thing using Redis. I started out with the recommendify ruby gem and then wrote my own, commendo. [http://rubygems.org/gems/commendo](http://rubygems.org/gems/commendo) We're using it for production pairwise comparison of about 30,000 resources at Meducation. Both visit-based similarity and content-based similarity. It implements Jaccard now and could be extended. hth, rob On 25 April 2014 23:09, NM [n.maisonneuve@gmail.com](mailto:n.maisonneuve@gmail.com) wrote: > I have N documents containing attributes. > > I needed to precompute a special similarity measure between each pairwise > of documents. > > Now I would to understand how to index and search using ES to answer a > query like > > "Retrieve me the Top N documents that are the most similar to document > ID 1 and having as fieldA = 1" > and facets the results according to a given field > > -- > > I was thinking to create an index of documnts with all the associated > pairwises as attributes,like: > > Doc > id: 1 > field1: 7 > field2: 10 > sim\_doc\_id2: 10 > sim\_doc\_id3: 8 > sim\_doc\_id4: 12 > ... > sim\_doc\_idN: 12 > > Doc > id: 2 > field1: 5 > field2: 2 > sim\_doc\_id1: 10 > sim\_doc\_id3: 3 > sim\_doc\_id4: 2 > ... > sim\_doc\_idN: 10 > .. > > issue with such design > The number of generated fields per document is very large for me (10K) > and I am not sure how to search efficiently (I tried a script score > like return doc['sim\_doc\_id1'] + field1) but it was quite slow.. > especially compared to a stupid loop in Java. however I would to use the > aggregation framework of ES to create facets of the results. > > Do you have any recommendation / guideline to handle this pb? > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAORoscnc2Py%2BKRDB0jED0M3%2B6vNPvuVekQz8WoL-eb7aj-Yy\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAORoscnc2Py%2BKRDB0jED0M3%2B6vNPvuVekQz8WoL-eb7aj-Yy_w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nicom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicom/32/488_2.png) [@nicom](https://discuss.elastic.co/u/nicom)\ **Post date:** [April 29, 2014, 8:22am UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210/3 "2014-04-29T08:22:46Z")
thanks Rob, might help but redis doesn't have the aggregation/facet features of ES that I would like to use. do someone else has some insight about solving this issue with Elasticsearch? Le samedi 26 avril 2014 00:25:23 UTC+2, Rob Styles a écrit : > Not elasticsearch, but I'm doing a similar thing using Redis. I started > out with the recommendify ruby gem and then wrote my own, commendo. > [commendo | RubyGems.org | your community gem host](http://rubygems.org/gems/commendo) > > We're using it for production pairwise comparison of about 30,000 > resources at Meducation. Both visit-based similarity and content-based > similarity. It implements Jaccard now and could be extended. > > hth, > > rob > > On 25 April 2014 23:09, NM \<[n.mais...@gmail.com](mailto:n.mais...@gmail.com) \\> wrote: > > > I have N documents containing attributes. > > > > I needed to precompute a special similarity measure between each > > pairwise of documents. > > > > Now I would to understand how to index and search using ES to answer a > > query like > > > > "Retrieve me the Top N documents that are the most similar to > > document ID 1 and having as fieldA = 1" > > and facets the results according to a given field > > > > -- > > > > I was thinking to create an index of documnts with all the associated > > pairwises as attributes,like: > > > > Doc > > id: 1 > > field1: 7 > > field2: 10 > > sim\_doc\_id2: 10 > > sim\_doc\_id3: 8 > > sim\_doc\_id4: 12 > > ... > > sim\_doc\_idN: 12 > > > > Doc > > id: 2 > > field1: 5 > > field2: 2 > > sim\_doc\_id1: 10 > > sim\_doc\_id3: 3 > > sim\_doc\_id4: 2 > > ... > > sim\_doc\_idN: 10 > > .. > > > > issue with such design > > The number of generated fields per document is very large for me (10K) > > and I am not sure how to search efficiently (I tried a script score > > like return doc['sim\_doc\_id1'] + field1) but it was quite slow.. > > especially compared to a stupid loop in Java. however I would to use the > > aggregation framework of ES to create facets of the results. > > > > Do you have any recommendation / guideline to handle this pb? > > > > Thanks > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4e5f22d6-4f0a-4739-92c8-8b2e85885a6f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/06075c2b-a1e9-45ed-9cd6-5bee49599bb3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/06075c2b-a1e9-45ed-9cd6-5bee49599bb3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Michael\_Sokolov](https://avatars.discourse-cdn.com/v4/letter/m/41988e/32.png) [@Michael\_Sokolov](https://discuss.elastic.co/u/Michael_Sokolov)\ **Post date:** [April 30, 2014, 2:02am UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210/4 "2014-04-30T02:02:20Z")
I've done something similar, but I use a single field for the related items; like this: id: 1 related: 2 3 9 100 category: xxx id: 2 related: 1 9 88 category: uuu xxx etc... You can limit the related items to the top N if you sort them by score when indexing and truncate the list. I don't model the relative scores in Lucene but you could do that in a gross way by repeating terms -Mike On Friday, April 25, 2014 6:09:13 PM UTC-4, NM wrote: > I have N documents containing attributes. > > I needed to precompute a special similarity measure between each pairwise > of documents. > > Now I would to understand how to index and search using ES to answer a > query like > > "Retrieve me the Top N documents that are the most similar to document > ID 1 and having as fieldA = 1" > and facets the results according to a given field > > -- > > I was thinking to create an index of documnts with all the associated > pairwises as attributes,like: > > Doc > id: 1 > field1: 7 > field2: 10 > sim\_doc\_id2: 10 > sim\_doc\_id3: 8 > sim\_doc\_id4: 12 > ... > sim\_doc\_idN: 12 > > Doc > id: 2 > field1: 5 > field2: 2 > sim\_doc\_id1: 10 > sim\_doc\_id3: 3 > sim\_doc\_id4: 2 > ... > sim\_doc\_idN: 10 > .. > > issue with such design > The number of generated fields per document is very large for me (10K) > and I am not sure how to search efficiently (I tried a script score > like return doc['sim\_doc\_id1'] + field1) but it was quite slow.. > especially compared to a stupid loop in Java. however I would to use the > aggregation framework of ES to create facets of the results. > > Do you have any recommendation / guideline to handle this pb? > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da3b3180-4d62-4119-87f0-6a415f72f0d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da3b3180-4d62-4119-87f0-6a415f72f0d0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:32am UTC](https://discuss.elastic.co/t/indexing-a-large-nx-n-matrix-of-similarity-with-es/17210/5 "2017-07-06T01:32:35Z")