# Duplicate logs in Logstash **URL:** \ **Category:** Logstash\ **Created:** [November 21, 2023, 11:52am UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630 "2023-11-21T11:52:12Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![marcowiskhy](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Post date:** [November 21, 2023, 11:52am UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/1 "2023-11-21T11:52:12Z")
I collect VPN logs through Logstash and index them in Elasticsearch, but I'm having the following problem: For each unique VPN connection (represented by TunnelID), there should be only one tunnel-up event and one tunnel-down event. However, sometimes the firewall duplicates the logs, so it can happen that a VPN connection has more than two assigned logs, as in the example below: ```auto @timestamp ConnectionDuration Fortinet Action TunnelID Nov 19, 2023 @ 06:56:35.552 993 tunnel-down 759607548 Nov 19, 2023 @ 06:56:35.552 993 tunnel-down 759607548 Nov 19, 2023 @ 06:56:35.551 993 tunnel-down 759607548 Nov 19, 2023 @ 06:40:02.412 - tunnel-up 759607548 Nov 19, 2023 @ 06:40:01.417 - tunnel-up 759607548 Nov 19, 2023 @ 06:06:00.507 24410 tunnel-down 759607532 Nov 19, 2023 @ 06:06:00.507 24410 tunnel-down 759607532 Nov 18, 2023 @ 23:19:13.928 0 tunnel-up 759607532 Nov 18, 2023 @ 23:19:09.898 0 tunnel-up 759607532 Nov 18, 2023 @ 23:19:08.841 13268 tunnel-down 759607512 Nov 18, 2023 @ 23:19:08.841 13268 tunnel-down 759607512 Nov 18, 2023 @ 19:38:06.661 0 tunnel-up 759607512 Nov 18, 2023 @ 19:37:59.677 0 tunnel-up 759607512 ``` I can use Logstash to avoid these "duplicates" so that they go to Elasticsearch as in the example below (for each unique TunnelID, there is only one tunnel-up event and one tunnel-down event)? ```auto @timestamp ConnectionDuration Fortinet Action TunnelID Nov 19, 2023 @ 06:56:35.551 993 tunnel-down 759607548 Nov 19, 2023 @ 06:40:01.417 - tunnel-up 759607548 Nov 19, 2023 @ 06:06:00.507 24410 tunnel-down 759607532 Nov 18, 2023 @ 23:19:09.898 0 tunnel-up 759607532 Nov 18, 2023 @ 23:19:08.841 13268 tunnel-down 759607512 Nov 18, 2023 @ 19:37:59.677 0 tunnel-up 759607512 ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 21, 2023, 5:22pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/2 "2023-11-21T17:22:12Z")
You could use a [fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) filter to generate a document\_id, so duplicated log entries will be overwritten. Combine the date, tunnel id, and action to create the fingerprint. ---
**Author:** ![marcowiskhy](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Post date:** [November 21, 2023, 5:32pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/3 "2023-11-21T17:32:57Z")
Hi, i tried this pipeline: ```auto input { file { path => "/etc/logstash/conf.d/vpn.json" sincedb_path => "/dev/null/" start_position => "beginning" } } filter { json { source => "message" } if [Fortinet Action] == "tunnel-up" or [Fortinet Action] == "tunnel-down" { fingerprint { source => ["TunnelID", "Fortinet Action"] target => "[@metadata][fingerprint]" method => "SHA256" } aggregate { task_id => "%{[@metadata][fingerprint]}" code => " map['@timestamp'] ||= event.get('@timestamp') map['ConnectionDuration'] ||= event.get('ConnectionDuration') map['Fortinet Action'] ||= event.get('Fortinet Action') map['TunnelID'] ||= event.get('TunnelID') " push_map_as_event_on_timeout => true timeout_task_id_field => "@timestamp" timeout => 300 } } } output { elasticsearch { hosts => ["https://192.168.xxx.xxx:9200"] index => "qradar-fortinetfw-1" action => "update" doc_as_upsert => true document_id => '%{[@metadata][fingerprint]}' } } ``` It worked, but it is not indexing all the logs, only the first connection log (TunnelID), the others seem to be ignored. What can it be? ---
**Author:** ![marcowiskhy](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Post date:** [November 21, 2023, 5:36pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/4 "2023-11-21T17:36:00Z")
What was expected: ```auto @timestamp ConnectionDuration Fortinet Action TunnelID Nov 19, 2023 @ 06:56:35.551 993 tunnel-down 759607548 Nov 19, 2023 @ 06:40:01.417 - tunnel-up 759607548 Nov 19, 2023 @ 06:06:00.507 24410 tunnel-down 759607532 Nov 18, 2023 @ 23:19:09.898 0 tunnel-up 759607532 Nov 18, 2023 @ 23:19:08.841 13268 tunnel-down 759607512 Nov 18, 2023 @ 19:37:59.677 0 tunnel-up 759607512 ``` What was indexed: ```auto @timestamp ConnectionDuration Fortinet Action TunnelID Nov 18, 2023 @ 23:19:08.841 13268 tunnel-down 759607512 Nov 18, 2023 @ 19:37:59.677 0 tunnel-up 759607512 ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 21, 2023, 5:39pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/5 "2023-11-21T17:39:35Z")
Your `source` in the `fingerprint` filter is an array, if you want to use both fields as the source for the fingerprint you need to also set [`concatenate_sources`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-concatenate_sources) to `true`. ---
**Author:** ![marcowiskhy](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Post date:** [November 23, 2023, 12:49pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/6 "2023-11-23T12:49:00Z")
Hi Leandro, thanks for reply, i'm a brazilian too. Your tip worked, but i'm receiving this error logs in some events: ```auto [2023-11-23T08:21:01,020][ERROR][logstash.javapipeline][qradar-fortinetfw-vpn] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"qradar-fortinetfw-vpn", :error=>"(TypeError) wrong argument type String (expected LogStash::Timestamp)", :exception=>Java::OrgJrubyExceptions::TypeError, :backtrace=>["RUBY.create_timeout_event(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-aggregate-2.10.0/lib/logstash/filters/aggregate.rb:278)", "RUBY.remove_expired_maps(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-aggregate-2.10.0/lib/logstash/filters/aggregate.rb:390)", "org.jruby.RubyHash.delete_if(org/jruby/RubyHash.java:2012)", "RUBY.remove_expired_maps(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-aggregate-2.10.0/lib/logstash/filters/aggregate.rb:385)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "RUBY.remove_expired_maps(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-aggregate-2.10.0/lib/logstash/filters/aggregate.rb:381)", "RUBY.flush(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-aggregate-2.10.0/lib/logstash/filters/aggregate.rb:327)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.flush(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:152)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#"} ``` I tried to use these settings: ```auto aggregate { task_id => "%{[@metadata][fingerprint]}" code => " map['@timestamp'] ||= LogStash::Timestamp.new(event.get('@timestamp')).to_time map['ConnectionDuration'] ||= event.get('ConnectionDuration') map['Fortinet Action'] ||= event.get('Fortinet Action') map['TunnelID'] ||= event.get('TunnelID') " push_map_as_event_on_timeout => true timeout_task_id_field => "@timestamp" timeout => 60 } ``` ```auto aggregate { task_id => "%{[@metadata][fingerprint]}" code => " map['@timestamp'] ||= LogStash::Timestamp.new(event.get('@timestamp')).to_iso8601 map['ConnectionDuration'] ||= event.get('ConnectionDuration') map['Fortinet Action'] ||= event.get('Fortinet Action') map['TunnelID'] ||= event.get('TunnelID') " push_map_as_event_on_timeout => true timeout_task_id_field => "@timestamp" timeout => 60 } ``` But not worked, some idea? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 23, 2023, 1:12pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/7 "2023-11-23T13:12:06Z")
> [@marcowiskhy](#): > > `(TypeError) wrong argument type String (expected LogStash::Timestamp)` It seems some issues in the aggregate filter related to the `@timestamp` field, but I'm not sure what because I do not use the aggregate filter. In fact, I'm not sure why you are using the aggregate filter, it seems that just having the fingerprint field would be enough to avoid duplicates. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 23, 2023, 5:20pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/8 "2023-11-23T17:20:37Z")
> [@marcowiskhy](#): > > `map['@timestamp'] ||= LogStash::Timestamp.new(event.get('@timestamp')).to_time` I, like Leandro, am puzzled why you are using aggregate, but to answer this question.... when aggregate creates the event (push\_map\_as\_event\_on\_timeout =\> true) it will create the [@timestamp] field from the map['@timestamp'] field, and that needs to be a LogStash::Timestamp, not a string. So changing this to ``` map['@timestamp'] ||= event.get('@timestamp') ``` will remove the error. ---
**Author:** ![marcowiskhy](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@marcowiskhy](https://discuss.elastic.co/u/marcowiskhy)\ **Post date:** [November 23, 2023, 6:15pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/9 "2023-11-23T18:15:47Z")
It was my mistake, I was reusing a pipeline that has some similar data. The tips from the gentlemen helped me and the problem was solved, thank you very much! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 21, 2023, 6:16pm UTC](https://discuss.elastic.co/t/duplicate-logs-in-logstash/347630/10 "2023-12-21T18:16:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.