# Deduplication filter? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 2, 2015, 4:50am UTC](https://discuss.elastic.co/t/deduplication-filter/21452 "2015-01-02T04:50:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Kshitij\_Gupta](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@Kshitij\_Gupta](https://discuss.elastic.co/u/Kshitij_Gupta)\ **Post date:** [January 2, 2015, 4:50am UTC](https://discuss.elastic.co/t/deduplication-filter/21452/1 "2015-01-02T04:50:37Z")
Hi, I am working on a system where I index multiple versions of objects by timestamp. I am including timestamp in the document id so that each version gets a different document id and is searchable by itself. This works fine. When I search over a time range, I get matching documents and I deduplicate the result set only keeping the latest version of each object from the result set. Now I would like to use aggregations in elasticsearch for building facets. But the facet calculation needs to happen after deduplication otherwise the counts will be inaccurate (objects for which multiple versions matched will be counted multiple times). Is there a deduplication filter available in elasticsearch? How do I write one myself? The only way I could get it to work was in multiple steps. First, query and get matching document ids. Then, deduplicate based on timestamp. Last, another query with ids filter listing all the ids from 2nd step to compute aggregations. But the last step takes even more time than the first query. Any suggestions? I was thinking that any system that tracks multiple versions of an object will face this issue. Thanks, Kshitij -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [January 2, 2015, 4:58am UTC](https://discuss.elastic.co/t/deduplication-filter/21452/2 "2015-01-02T04:58:29Z")
Simplest way might be to push an update to the old versions of the documents to mark them as old and do aggregations filtering those out. There isn't a great way to deduplicate, really. On Thu, Jan 1, 2015 at 11:50 PM, Kshitij Gupta [kshitij@vnera.com](mailto:kshitij@vnera.com) wrote: > Hi, > > I am working on a system where I index multiple versions of objects by > timestamp. I am including timestamp in the document id so that each version > gets a different document id and is searchable by itself. This works fine. > When I search over a time range, I get matching documents and I deduplicate > the result set only keeping the latest version of each object from the > result set. > > Now I would like to use aggregations in elasticsearch for building facets. > But the facet calculation needs to happen after deduplication otherwise the > counts will be inaccurate (objects for which multiple versions matched will > be counted multiple times). Is there a deduplication filter available in > elasticsearch? How do I write one myself? > > The only way I could get it to work was in multiple steps. First, query > and get matching document ids. Then, deduplicate based on timestamp. Last, > another query with ids filter listing all the ids from 2nd step to compute > aggregations. But the last step takes even more time than the first query. > > Any suggestions? I was thinking that any system that tracks multiple > versions of an object will face this issue. > > Thanks, > Kshitij > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 2, 2015, 8:13am UTC](https://discuss.elastic.co/t/deduplication-filter/21452/3 "2015-01-02T08:13:36Z")
The most scalable method is to index all the documents in timewindow based indexes and use the latest timewindow index only so no deduplication is required. Jörg On Fri, Jan 2, 2015 at 5:58 AM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > Simplest way might be to push an update to the old versions of the > documents to mark them as old and do aggregations filtering those out. > There isn't a great way to deduplicate, really. > > On Thu, Jan 1, 2015 at 11:50 PM, Kshitij Gupta [kshitij@vnera.com](mailto:kshitij@vnera.com) wrote: > > > Hi, > > > > I am working on a system where I index multiple versions of objects by > > timestamp. I am including timestamp in the document id so that each version > > gets a different document id and is searchable by itself. This works fine. > > When I search over a time range, I get matching documents and I deduplicate > > the result set only keeping the latest version of each object from the > > result set. > > > > Now I would like to use aggregations in elasticsearch for building > > facets. But the facet calculation needs to happen after deduplication > > otherwise the counts will be inaccurate (objects for which multiple > > versions matched will be counted multiple times). Is there a deduplication > > filter available in elasticsearch? How do I write one myself? > > > > The only way I could get it to work was in multiple steps. First, query > > and get matching document ids. Then, deduplicate based on timestamp. Last, > > another query with ids filter listing all the ids from 2nd step to compute > > aggregations. But the last step takes even more time than the first query. > > > > Any suggestions? I was thinking that any system that tracks multiple > > versions of an object will face this issue. > > > > Thanks, > > Kshitij > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ptmjLZW8Jfuhp%2BPQSekrLwNZAYN%2BhuMyv8BnUT6i-dQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZ1Us41i%3DzqqA0Dv3i%2BtSyDHZKb7QoS9-RqhtGtRxFqg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZ1Us41i%3DzqqA0Dv3i%2BtSyDHZKb7QoS9-RqhtGtRxFqg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kshitij\_Gupta](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@Kshitij\_Gupta](https://discuss.elastic.co/u/Kshitij_Gupta)\ **Post date:** [January 2, 2015, 12:20pm UTC](https://discuss.elastic.co/t/deduplication-filter/21452/4 "2015-01-02T12:20:40Z")
Good suggestion and we do that already. But we allow querying over a time range as well which could be in the past. Here multiple "old" versions might match and we need to pick the latest "old" version. On Friday, January 2, 2015 10:28:58 AM UTC+5:30, Nikolas Everett wrote: > Simplest way might be to push an update to the old versions of the > documents to mark them as old and do aggregations filtering those out. > There isn't a great way to deduplicate, really. > > On Thu, Jan 1, 2015 at 11:50 PM, Kshitij Gupta \<[ksh...@vnera.com](mailto:ksh...@vnera.com) > \\> wrote: > > > Hi, > > > > I am working on a system where I index multiple versions of objects by > > timestamp. I am including timestamp in the document id so that each version > > gets a different document id and is searchable by itself. This works fine. > > When I search over a time range, I get matching documents and I deduplicate > > the result set only keeping the latest version of each object from the > > result set. > > > > Now I would like to use aggregations in elasticsearch for building > > facets. But the facet calculation needs to happen after deduplication > > otherwise the counts will be inaccurate (objects for which multiple > > versions matched will be counted multiple times). Is there a deduplication > > filter available in elasticsearch? How do I write one myself? > > > > The only way I could get it to work was in multiple steps. First, query > > and get matching document ids. Then, deduplicate based on timestamp. Last, > > another query with ids filter listing all the ids from 2nd step to compute > > aggregations. But the last step takes even more time than the first query. > > > > Any suggestions? I was thinking that any system that tracks multiple > > versions of an object will face this issue. > > > > Thanks, > > Kshitij > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11f983e9-b661-4ae6-83d1-6875bedd8c7e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7264abb-fe9b-48d7-ae8c-2b98d33f12d8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7264abb-fe9b-48d7-ae8c-2b98d33f12d8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:41am UTC](https://discuss.elastic.co/t/deduplication-filter/21452/5 "2017-07-06T00:41:10Z")