# How to force entries to be unique **URL:** \ **Category:** Elasticsearch\ **Created:** [July 26, 2018, 2:03pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787 "2018-07-26T14:03:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![mivipe](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mivipe](https://discuss.elastic.co/u/mivipe)\ **Post date:** [July 26, 2018, 2:03pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/1 "2018-07-26T14:03:36Z")
Dear experts, I have a simple Filebeat-Elasticsearch-Kibana configuration. In my logs every entry has a uniqueID field. Everything seems to work fine until, for some reason, all entries become duplicated. I've noticed that the field \_id is actually different in each of the duplicates so I've tried using a pipeline with the set processor to overwrite \_id with the value of my uniqueID. I was hoping ES would just update/overwrite entries when \_id already exists but no luck... I've also tried to overwrite the field \_uid, but it seems I am not allowed. Do you have any suggestion? Regards ---
**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\ **Post date:** [July 26, 2018, 5:41pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/2 "2018-07-26T17:41:11Z")
Can you share your pipeline and what the documents look like before/after transformation? ---
**Author:** ![mivipe](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mivipe](https://discuss.elastic.co/u/mivipe)\ **Post date:** [July 27, 2018, 12:49pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/3 "2018-07-27T12:49:28Z")
Here is a simple tests using \_simulate ``` POST _ingest/pipeline/_simulate { "pipeline": { "processors": [ { "date": { "field": "CompletionDate", "target_field": "@timestamp", "formats": [ "UNIX_MS" ], "timezone": "Europe/Amsterdam" }, "set": { "field": "_id", "value": "{{GlobalJobId}}" } } ] }, "docs": [ { "_index": "index", "_type": "type", "_id": "1", "_version": 4, "found": true, "_source": { "Owner": "owner1", "GlobalJobId": "halley#2213.99#1532617316", "JobCurrentStartDate": "1532617485000", "CompletionDate": "1532617496000", "UsedTime": 11, "UsedCpu": 0, "MemoryUsage": 1 } } ] } ``` and the result ``` { "docs": [ { "doc": { "_index": "index", "_type": "type", "_id": "halley#2213.99#1532617316", "_source": { "Owner": "owner1", "GlobalJobId": "halley#2213.99#1532617316", "@timestamp": "2018-07-26T17:04:56.000+02:00", "UsedCpu": 0, "JobCurrentStartDate": "1532617485000", "UsedTime": 11, "CompletionDate": "1532617496000", "MemoryUsage": 1 }, "_ingest": { "timestamp": "2018-07-27T12:37:29.904Z" } } } ] } ``` Regards ---
**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\ **Post date:** [July 30, 2018, 12:29am UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/4 "2018-07-30T00:29:15Z")
That looks like it's working: `_id` looks like it's 1 going in and it's `halley#2213.99#1532617316` coming out. Are you saying you're seeing duplicate `_id` of values of `halley#2213.99#1532617316` ---
**Author:** ![mivipe](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mivipe](https://discuss.elastic.co/u/mivipe)\ **Post date:** [July 30, 2018, 9:04am UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/5 "2018-07-30T09:04:41Z")
Exactly. It works fine for a few days but at some point I start seeing duplicate \_id values. ---
**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\ **Post date:** [July 30, 2018, 7:40pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/6 "2018-07-30T19:40:11Z")
Are you maybe seeing duplicate \_ids across different indices? ---
**Author:** ![mivipe](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mivipe](https://discuss.elastic.co/u/mivipe)\ **Post date:** [July 30, 2018, 8:31pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/7 "2018-07-30T20:31:21Z")
As far as I understand, I only have one index. I tried to change the least possible from the default configuration. How can I rule that out? ---
**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\ **Post date:** [July 30, 2018, 9:24pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/8 "2018-07-30T21:24:27Z")
You said in the beginning: > all entries become duplicated Can you show a few examples of these duplicates and a search that produces them? ---
**Author:** ![mivipe](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mivipe](https://discuss.elastic.co/u/mivipe)\ **Post date:** [July 31, 2018, 7:58am UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/9 "2018-07-31T07:58:43Z")
Here is an example. The only difference is indeed the field "\_index". I thought including the date in the name of the index was recommended and different dates did not count as different indices. I'll test using a static name. ``` GET condor-*/_search { "query": { "match": { "_id": { "query": "halley#2213.98#1532617316" } } } } ``` Then, the result, ``` { "took": 2, "timed_out": false, "_shards": { "total": 20, "successful": 20, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": 1, "hits": [ { "_index": "condor-2018.07.26", "_type": "doc", "_id": "halley#2213.98#1532617316", "_score": 1, "_source": { "GlobalJobId": "halley#2213.98#1532617316", "Owner": "owner1", "offset": 1786491, "UsedCpu": 0, "JobCurrentStartDate": "1532617484000", "input_type": "log", "source": "/var/log/condor_history.log", "type": "log", "MemoryUsage": 1, "@timestamp": "2018-07-26T17:04:56.000+02:00", "UsedTime": 12, "beat": { "hostname": "halley.fisica.unimi.it", "name": "halley.fisica.unimi.it", "version": "5.6.9" }, "CompletionDate": "1532617496000" } }, { "_index": "condor-2018.07.27", "_type": "doc", "_id": "halley#2213.98#1532617316", "_score": 1, "_source": { "GlobalJobId": "halley#2213.98#1532617316", "Owner": "owner1", "offset": 1786491, "UsedCpu": 0, "JobCurrentStartDate": "1532617484000", "input_type": "log", "source": "/var/log/condor_history.log", "type": "log", "MemoryUsage": 1, "@timestamp": "2018-07-26T17:04:56.000+02:00", "UsedTime": 12, "beat": { "hostname": "halley.fisica.unimi.it", "name": "halley.fisica.unimi.it", "version": "5.6.9" }, "CompletionDate": "1532617496000" } }, { "_index": "condor-2018.07.28", "_type": "doc", "_id": "halley#2213.98#1532617316", "_score": 1, "_source": { "GlobalJobId": "halley#2213.98#1532617316", "Owner": "owner1", "offset": 1786491, "UsedCpu": 0, "JobCurrentStartDate": "1532617484000", "input_type": "log", "source": "/var/log/condor_history.log", "type": "log", "MemoryUsage": 1, "@timestamp": "2018-07-26T17:04:56.000+02:00", "UsedTime": 12, "beat": { "hostname": "halley.fisica.unimi.it", "name": "halley.fisica.unimi.it", "version": "5.6.9" }, "CompletionDate": "1532617496000" } } ] } } ``` ---
**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\ **Post date:** [July 31, 2018, 9:34pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/10 "2018-07-31T21:34:02Z")
> I thought including the date in the name of the index was recommended and different dates did not count as different indices. Generally, including the date in the name of the index for timeseries data _is_ recommended (or using [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html) with its numeric postfix). One of the reasons for that is that you're probably going to want to delete some data at some point and Elasticsearch is _way_ more efficient at deleting entire indices than "delete by query." So when the data from 2018-07-26 is no longer useful to your business, you just delete the entire index. Also, if your queries have times in them, Elasticsearch can quickly rule out indices/shards that can't possibly match the date of the query. Anyway, Elasticsearch IDs are only unique to a single index. You're talking about going to a single index to try to solve this, but another solution is to make sure the same ID always goes into the same index. That is, rather than using the _current_ date at the end of the index, you could use something like `CompletionDate`. There's even an [ingest node processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-index-name-processor.html) to do this for you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2018, 9:37pm UTC](https://discuss.elastic.co/t/how-to-force-entries-to-be-unique/141787/11 "2018-08-28T21:37:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.