# Implementing set logic on an array field **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2014, 8:12am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18549 "2014-07-09T08:12:54Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Nikojiro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikojiro/32/1414_2.png) [@Nikojiro](https://discuss.elastic.co/u/Nikojiro)\ **Post date:** [July 9, 2014, 8:12am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18549/1 "2014-07-09T08:12:54Z")
Hi, I am very often confronted to the following pattern when doing batch indexing in update mode. Let me picture it with a short example: Let's pretend I am indexing social bookmarking data: a very crude document would be something like: _{ "url": "[http://somecoolwebsite.org](http://somecoolwebsite.org) [http://somecoolwebsite.org/](http://somecoolwebsite.org/)", ... "usertags": ["tag1", "tag2", ..., "tagN"] ... }_ My indexer processes a very large list of user bookmarks, and batch updates/upserts the document in Elasticsearch. My problem is that if I simply use concatenation in the update script, I may end up with lots of duplicate values in my _usertag_ array, as many users potentially used the same tag over and over again on a given url. Instead I would like to have a set logic on the array values. Currently I have this pattern on a bunch of uses cases, and I generally handle that within the batch program by deduplicating values, and using a BerkeleyDB to have as much data as I can in memory. However the performance cost becomes prohibitive when I have to perform set logic over millions of input records. Below 5M I manage to have an acceptable cost, but past 5M insertion time in my BDB becomes unacceptable. Another way would be not to deduplicate and to use terms facet at query time to obtain deduplicated values, bit the index size will potentially grow significantly. Lastly one could put in place a post-processing batch to deduplicate the values, but that sums up tp reindexing everything. Using batch treatment and parallel execution this could probably scale pretty well, but would be time consuming. This is probably a very common pattern, however I'd very much appreciate to have some pointers on how other Elasticsearch users dealt with it. Best regards, Nicolas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [July 9, 2014, 6:26pm UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18549/2 "2014-07-09T18:26:35Z")
Hello Nicolas , Why don't you handle this during the update itself. Update can be done using a script. So something like - curl -XPOST 'localhost:9200/test/type1/1/\_update' -d '{ "script" : "ctx.\_source.tags.contains(tag) ? true : ctx.\_source.tags += tag", ``` "params" : { "tag" : "blue" } ``` }' Link - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Thanks Vineeth On Wed, Jul 9, 2014 at 1:42 PM, Nicolas Giraud [nicosensei@gmail.com](mailto:nicosensei@gmail.com) wrote: > Hi, > > I am very often confronted to the following pattern when doing batch > indexing in update mode. Let me picture it with a short example: > > Let's pretend I am indexing social bookmarking data: a very crude document > would be something like: > > _{ "url": "[http://somecoolwebsite.org](http://somecoolwebsite.org) [http://somecoolwebsite.org/](http://somecoolwebsite.org/)", > ... "usertags": ["tag1", "tag2", ..., "tagN"] ... }_ > > My indexer processes a very large list of user bookmarks, and batch > updates/upserts the document in Elasticsearch. My problem is that if I > simply use concatenation in the update script, I may end up with lots of > duplicate values in my _usertag_ array, as many users potentially used > the same tag over and over again on a given url. Instead I would like to > have a set logic on the array values. > > Currently I have this pattern on a bunch of uses cases, and I generally > handle that within the batch program by deduplicating values, and using a > BerkeleyDB to have as much data as I can in memory. However the performance > cost becomes prohibitive when I have to perform set logic over millions > of input > records. Below 5M I manage to have an acceptable cost, but past 5M > insertion time in my BDB becomes unacceptable. > > Another way would be not to deduplicate and to use terms facet at query > time to obtain deduplicated values, bit the index size will potentially > grow significantly. > > Lastly one could put in place a post-processing batch to deduplicate the > values, but that sums up tp reindexing everything. Using batch treatment > and parallel execution this could probably scale pretty well, but would be > time consuming. > > This is probably a very common pattern, however I'd very much appreciate > to have some pointers on how other Elasticsearch users dealt with it. > > Best regards, > > Nicolas > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4d8a3512-08d2-4783-b1d2-622d2b546502%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DESbSDBjm0f1N6o\_LrGPaLT2-0mzPut%3De8zzgtDJQEDg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DESbSDBjm0f1N6o_LrGPaLT2-0mzPut%3De8zzgtDJQEDg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18549/3 "2017-07-06T01:16:57Z")