# Implementing set logic on an array field

**URL:** <https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18524>\
**Category:** Elasticsearch\
**Created:** [July 8, 2014, 9:14am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18524 "2014-07-08T09:14:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nikojiro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikojiro/32/1414_2.png) [@Nikojiro](https://discuss.elastic.co/u/Nikojiro)\
**Post date:** [July 8, 2014, 9:14am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18524/1 "2014-07-08T09:14:57Z")

</div>

Hi,

I am very often confronted to the following pattern when doing batch  
indexing un update mode. Let me picture it with a short example:

Let's pretend I am indexing social bookmarking data: a very crude document  
would be something like:

/{  
"url": "[http://somecoolwebsite.org](http://somecoolwebsite.org)",  
...  
"usertags": ["tag1", "tag2", ..., "tagN"]  
...  
}/

My indexer processes a very large list of user bookmarks, and batch  
updates/upserts the document in Elasticsearch. My problem is that if I  
simply use concatenation in the update script, I may end up with lots of  
duplicate values in my /usertag/ array, as many users potentially used the  
same tag over and over again on a given url. Instead I would like to have a  
set logic on the array values.

Currently I have this pattern on a bunch of uses cases, and I generally  
handle that within the batch program by deduplicating values, and using a  
BerkeleyDB to have as much data as I can in memory. However the performance  
cost becomes prohibitive when I have to perform set logic over millions of  
input  
records. Below 5M I manage to have an acceptable cost, but past 5M insertion  
time in my BDB becomes unacceptable.

Another way would be not to deduplicate and to use terms facet at query time  
to obtain deduplicated values, bit the index size will potentially grow  
significantly.

Lastly one could put in place a post-processing batch to deduplicate the  
values, but that sums up tp reindexing everything. Using batch treatment and  
parallel execution this could probably scale pretty well, but would be time  
consuming.

This is probably a very common pattern, however I'd very much appreciate to  
have some pointers on how other Elasticsearch users dealt with it.

Best regards,

Nicolas

--  
View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Implementing-set-logic-on-an-array-field-tp4059420.html](http://elasticsearch-users.115913.n3.nabble.com/Implementing-set-logic-on-an-array-field-tp4059420.html)  
Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1404810897523-4059420.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1404810897523-4059420.post%40n3.nabble.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:17am UTC](https://discuss.elastic.co/t/implementing-set-logic-on-an-array-field/18524/2 "2017-07-06T01:17:22Z")

</div>