# Fingerprint duplicate hash for different events **URL:** \ **Category:** Logstash\ **Created:** [August 25, 2021, 9:07am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462 "2021-08-25T09:07:04Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Post date:** [August 25, 2021, 9:07am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462/1 "2021-08-25T09:07:04Z")
Hi, I have a strange issue which came to light after starting to use datastreams (and thus create events instead of updates). The following fingerprint config we have in logstash: ```auto ### Add a fingerprint to prevent duplicate log events. fingerprint { concatenate_sources => true source => ["message","agent.hostname"] target => "[@metadata][fingerprint]" method => "SHA1" key => "deduplication-key" } ``` In Elastic This event exists: ```auto { "_index": ".ds-agl-api-ds-2021.08.25-001447", "_type": "_doc", "_id": "c3b600da568b237e44e68f1a5bd718246e58a908", "_score": 1, "_source": { "input": { "type": "log" }, "ecs": { "version": "1.6.0" }, "log-message": "Configuration cache updated!", "tags": [ "avs6", "api-log", "apigateway", "asd", "beats_input_codec_plain_applied" ], "log-level": "INFO", "message": "ts: 2021-08-25 10:36:40.769 | logLevel: INFO | appId: AGL | thread: | SID: undefined | TN: undefined | clientIp: undefined | userId: ANONYMOUS | apiType: NANO | api: | platform: | eventType: NONE | message: Configuration cache updated!", "log": { "file": { "path": "/product/AGL/agl-core/logs/agl.log" }, "offset": 3737176171 }, "api-type": "NANO", "@version": "1", "fields": { "environment": "production" }, "@timestamp": "2021-08-25T08:36:40.769Z", "app-id": "AGL", "user-id": "ANONYMOUS", "agent": { "name": "papps1443.prdl.itv.local", "ephemeral_id": "1daa9993-bf4e-4ce0-bc00-bb3762c88820", "version": "7.10.2", "hostname": "papps1443.prdl.itv.local", "id": "139e78fb-a5d6-47f9-813d-7d63e08b5d32", "type": "filebeat" }, "event-type": "NONE" }, "fields": { "log-message": [ "Configuration cache updated!" ], "app-id": [ "AGL" ], "api-type": [ "NANO" ], "event-type": [ "NONE" ], "user-id": [ "ANONYMOUS" ], "input.type": [ "log" ], "log.offset": [ 3737176171 ], "fields.environment": [ "production" ], "agent.hostname": [ "papps1443.prdl.itv.local" ], "message": [ "ts: 2021-08-25 10:36:40.769 | logLevel: INFO | appId: AGL | thread: | SID: undefined | TN: undefined | clientIp: undefined | userId: ANONYMOUS | apiType: NANO | api: | platform: | eventType: NONE | message: Configuration cache updated!" ], "tags": [ "avs6", "api-log", "apigateway", "asd", "beats_input_codec_plain_applied" ], "agent.type": [ "filebeat" ], "@timestamp": [ "2021-08-25T08:36:40.769Z" ], "agent.id": [ "139e78fb-a5d6-47f9-813d-7d63e08b5d32" ], "ecs.version": [ "1.6.0" ], "log-level": [ "INFO" ], "log.file.path": [ "/product/AGL/agl-core/logs/agl.log" ], "@version": [ "1" ], "agent.ephemeral_id": [ "1daa9993-bf4e-4ce0-bc00-bb3762c88820" ], "agent.name": [ "papps1443.prdl.itv.local" ], "agent.version": [ "7.10.2" ] } } ``` In my logstash log, this error exists: ```auto [2021-08-25T10:36:42,590][WARN][logstash.outputs.elasticsearch] Failed action { :status=>409, :action=>[ "create", { :_id=>"c3b600da568b237e44e68f1a5bd718246e58a908", :_index=>"agl-api-ds", :routing=>nil }, { "input"=>{ "type"=>"log" }, "ecs"=>{ "version"=>"1.6.0" }, "log-message"=>"Configuration cache updated!", "tags"=>[ "avs6", "api-log", "apigateway", "asd", "beats_input_codec_plain_applied" ], "log-level"=>"INFO", "message"=>"ts: 2021-08-25 10:36:40.769 | logLevel: INFO | appId: AGL | thread: | SID: undefined | TN: undefined | clientIp: undefined | userId: ANONYMOUS | apiType: NANO | api: | platform: | eventType: NONE | message: Configuration cache updated!", "log"=>{ "file"=>{ "path"=>"/product/AGL/agl-core/logs/agl.log" }, "offset"=>2101471568 }, "api-type"=>"NANO", "@version"=>"1", "fields"=>{ "environment"=>"production" }, "@timestamp"=>2021-08-25T08: 36: 40.769Z, "app-id"=>"AGL", "user-id"=>"ANONYMOUS", "agent"=>{ "name"=>"papps1632.prdl.itv.local", "ephemeral_id"=>"740640a2-daed-4807-9ff6-55bfdabeb066", "version"=>"7.10.2", "hostname"=>"papps1632.prdl.itv.local", "id"=>"b0d97044-dc4d-4778-8a63-a64368a9b26c", "type"=>"filebeat" }, "event-type"=>"NONE" } ], :response=>{ "create"=>{ "_index"=>".ds-agl-api-ds-2021.08.25-001447", "_type"=>"_doc", "_id"=>"c3b600da568b237e44e68f1a5bd718246e58a908", "status"=>409, "error"=>{ "type"=>"version_conflict_engine_exception", "reason"=>"[c3b600da568b237e44e68f1a5bd718246e58a908]: version conflict, document already exists (current version [1])", "index_uuid"=>"1U0Gff1CQVab4aaDbTYqLQ", "shard"=>"0", "index"=>".ds-agl-api-ds-2021.08.25-001447" } } } } ``` Please consider the agent hostname in both situations. Both messages came from a different server. How is it possible this generates the same ID? ---
**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Post date:** [August 26, 2021, 11:35am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462/2 "2021-08-26T11:35:35Z")
With some testing it seems that the agent.hostname (or agent.name if we use that one, simply is not added to the string to be hashed. Does the concatenate\_source parameter even work? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 26, 2021, 3:33pm UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462/3 "2021-08-26T15:33:45Z")
> [@Tuckson](#): > > `source => ["message","agent.hostname"]` The event has no agent.hostname field, it has an [agent][hostname] field. If a field does not exist it is not included in the fingerprint. ---
**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Post date:** [September 7, 2021, 10:51am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462/4 "2021-09-07T10:51:11Z")
Thnx. Looks like this indeed was the solution. Very confusing that the dot notation is mentioned on the concerning documentation pages though. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 5, 2021, 10:52am UTC](https://discuss.elastic.co/t/fingerprint-duplicate-hash-for-different-events/282462/5 "2021-10-05T10:52:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.