# Retrieving document groups with MoreLikeThis query **URL:** \ **Category:** Elasticsearch\ **Created:** [July 24, 2014, 9:03pm UTC](https://discuss.elastic.co/t/retrieving-document-groups-with-morelikethis-query/18880 "2014-07-24T21:03:17Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Alain\_Desilets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alain_desilets/32/29314_2.png) [@Alain\_Desilets](https://discuss.elastic.co/u/Alain_Desilets)\ **Post date:** [July 24, 2014, 9:03pm UTC](https://discuss.elastic.co/t/retrieving-document-groups-with-morelikethis-query/18880/1 "2014-07-24T21:03:17Z")
Say I have a set of documents D={d1,..., dn} indexed in ES. I also have a set of document groups G={g1,...,gk}, where the groups may not be disjoint. Given a group g in G, called the Query Group, I want to find "similar" groups g' in G, called the Similar Groups. where "similar" means that the documents in g tend to use the same words as documents in each of the g'. What would be the easiest way do this with ES? Note that I already know how to find individual documents d which are similar to a group g. You just do a MoreLikeThis search, feeding it a multi-get query with the IDs of the various documents in g. But this will retrieve individual documents, when in fact what I want document groups. I can think of several ways of doing this (see below). Not sure what the pros and cons of each approach are, in terms of speed and "accuracy". Does anyone have advice on this? Thx (PS Sorry for the long post, but it's a pretty complex issue) Alain === Approach 1: Groups as pseudo-documents that concatenate content of their members Create a type 'documents-group'. These are "pseudo-documents" whose content will be the concatenation of the content of all documents in a given group. You then do a MoreLikeThis (MLT) query, feeding it the pseudo-document for the Query Group, and asking specifically for documents of type documents-group PROS: - Can be supported for sure. CONS: - Need to write code in the application to create and manage those pseudo-documents. - The content of each document will be duplicated N times, where N is the number of groups that it belongs to. -- This means the index will be much larger (in my application, a document can belong to hundreds of groups) - Requires a lot of reindexing -- Everytime you change a document, you need to re-index all the groups it belongs to. -- If you add/remove a document to a group, you need to re-index that group, which means reindex the equivalent of all the documents it contains (in my application, a group may contain 100 documents or more) === Approach 2: Groups as pseudo-documents containing only the most important terms This is similar to approach 1, except that the pseudo-document contains a list of the most significant terms of documents in that group (obtained with the Significant Terms aggregation), as opposed to containing a concatenation of all the member documents. You then do a MoreLikeThis (MLT) query, feeding it the documents-group document for the Query Group, and asking specifically for documents of type documents-group PROS: - Can be supported for sure. - The content of individual documents is not completely duplicated. At words, a handful of terms from each document may be duplicated. CONS: - Need to write code in the application to create and manage those document-group pseudo-documents. - Requires a lot of reindexing -- Everytime you change a document, you need to re-generate the the list of significant terms for all the groups it belongs to. -- If you add/remove a document to a group, you re-generate the the list of significant terms for that group. - May be less accurate, as similarity is based on only the most significant terms, instead of all available terms. In other words, it could be that terms which seem a-priori unimportant, turn out to be very important when trying to determine similarity to the Query Group. === Approach 3: Multi-get MLT, with automatic aggregation of average score on the groups Each document includes a field that specifies the groups it belongs to. You then do a MLT search, feeding it a Multi-Get query of the IDs of all the documents in the Query Group. The query also asks ES to do a metric aggregate using the Group membership field as the basis for the aggregation, and the average MLT score as the metric to be computed. The group that has the highest MLT average score is considered to be the most relevant group. PROS: - No need to write application code to create and manage pseudo-documents for the groups. - Does not requires too much reindexing -- If you modify a document, you only need to reindex that document. -- If you add/remove a document in a group, you only need to reindex that one document (because the list of groups that the document belongs to has changed). CONS: - Not clear that it can be done. Possible problems are: -- The groups are overlapping. In other words, a given document may appear in more than one parts of the aggregation. Not sure if ES aggregation deals properly with that. -- Not sure that you can use the MLT score as the metric to be averaged. The reason being that the score is a dynamic property of each document, whose value is determined specifically by MLT at query time. Metric aggregation is typically used with static attributes (ex: a document's author). === Approach 4: Multi-get MLT, with "manual" aggregation of average score on the groups This is very similar to Approach 2, except that the aggregation of MLT scores is done by the application instead of ES. You do a MLT search, feeding it a Multi-Get query of the IDs of all the documents in the Query Group, and asking ES to give you up to say, 10K individual similar documents. For each group g in G, you then compute a relevance score by looking at the 10K hits. For example, you may take from those 10K hits, the ones that belong to a given group, add their scores, and divide that sum by the total number of documents in the group (because larger groups are a-priori more likely to contain a hit in the top 10K most similar documents). PROS: - Definitely doable - Does not require too much reindexing -- If you modify a document, you only need to reindex that document. -- If you add/remove a document in a group, you only need to reindex that one document (because the list-of-groups attribute for that document will have changed) CONS: - Need to write application code to do the aggregation. - Not clear that the aggregation will always work well -- In particular, not clear how to normalize for the fact that larger groups will tend to include more hits in the 10K most similar documents. === Approach 5: Create the groups as parents of their documents Create a new type 'documents-group'. Create a parent-child relationship between a group and each of the documents it contains. When searching, do a MLT on the parent document that represents the Query Group, and ask for it to return documents that are of type 'documents-group'. PROS: - No need to write application code to create pseudo-documents for the groups. - Fast (Probably): -- If you modify a document, you only need to reindex that document -- If you add/remove a document in a group, you probably don't end up reindexing anything CONS: - Not clear that it can be done. Possible problems are: -- Not clear that an ES document can have more than one parents (and our groups are overlapping) -- When you do a MTL on a parent, does it actually consider the content of every child it contains? -- Similarly, when evaluating the similarity of various parent documents, does it consider the content of every one of its children? -- Not really sure what actually happens in terms of reindexing if you you modify the list of children, or the content of one of the children. It could be that under the hood, ES keeps some kind of duplicate index in the parent, and that this duplicate needs to be updated when you change one of the children. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da88c17d-40a9-4083-9b79-cda816610d50%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da88c17d-40a9-4083-9b79-cda816610d50%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:13am UTC](https://discuss.elastic.co/t/retrieving-document-groups-with-morelikethis-query/18880/2 "2017-07-06T01:13:19Z")