# Duplicate logs - logstash-input-s3? **URL:** \ **Category:** Logstash\ **Created:** [January 14, 2016, 2:37pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231 "2016-01-14T14:37:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\ **Post date:** [January 14, 2016, 2:37pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/1 "2016-01-14T14:37:27Z")
Hi all, I'm using LS2.1.1 with ES2.1.1, in AWS, to log various sources within my environment. One of these sources is AWS CloudTrail. I've noticed a strange problem when trying to optimise my elasticsearch cluster, by reducing the indices and shards, more details [here](https://discuss.elastic.co/t/please-help-es-2-1-1-cluster-randomly-crashing/38861/12). So I've optimised my cluster by reducing both my total indices and total shards. Good news so far. My input used for logstash-input-s3: `input { s3 { bucket => "my-logs-cloudtrail" delete => false interval => 60 # seconds prefix => "AWSLogs/MYACCOUNTID/CloudTrail/" type => "cloudtrail" codec => "cloudtrail" credentials => "/etc/logstash/s3_credentials.ini" sincedb_path => "/opt/logstash_cloudtrail/MYACCOUNTID-sincedb" } }` My filter used to parse the incoming logs: `filter { if [userIdentity][accountId] =~ "MYACCOUNTID" { mutate { add_field => ["accountName", "mylogs-aws-development"] } } }` My output used to send to Elasticsearch: `output { if [type] == "cloudtrail" { elasticsearch { hosts => "MYESCLUSTER" index => "logstash-cloudtrail" } } else { elasticsearch { hosts => "MYESCLUSTER" index => "wtf-are-these-logs" } } stdout { codec => "rubydebug" } }` I noticed that my filter is working as, as it was before, however, _ **something odd started to happen:** _ ![](https://us1.discourse-cdn.com/elastic/original/2X/7/76445ff4ed3ef238b0f3385210eb1681b20cae20.png) -Document count continues to grow at nearly the same rate. -I restarted the cluster logging a few times. -I triple checked my input, filter, and output. Then I started to compare events from each index, and noticed: -Sample events were identical. -Events in `wtf-are-these-logs` index are also labelled `type: cloudtrail` -The frequency of events, displayed using kibana, is almost identical. _ **I cant explain this.** _ Has anyone else seen this before? Can anyone assist with an explanation or how I can stop this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 15, 2016, 6:53am UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/2 "2016-01-15T06:53:23Z")
Make sure you don't have any extra files in /etc/logstash/conf.d. Logstash will read every single file and effectively concatenate them. ---
**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\ **Post date:** [January 15, 2016, 9:44am UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/3 "2016-01-15T09:44:20Z")
thanks @magnusbaeck, I thought to check that right away. Does it matter that my `input` and `filter` were in the same file, and that my `output` in a separate file? The current file structure I use is: `10-inputfile.conf` (contains my input and filters) `30-outputfile.conf` (contains my output configuration) I have **just now** modified this to look like (How it was configured sometime ago): `10-inputfile.conf` (contains my input configuration) `20-filterfile.conf` (contains my filter configuration) `30-outputfile.conf` (contains my output configuration) I have considered putting the entire `input` =\> `filter` =\> `output` pipeline in the same file, but I am unsure if this makes any difference. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 15, 2016, 9:49am UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/4 "2016-01-15T09:49:42Z")
The only thing that matters is the internal order of all filters. Having multiple files in a directory is exactly equivalent to doing `cat /etc/logstash/conf.d/* > logstash.conf` and pointing Logstash to logstash.conf. ---
**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\ **Post date:** [January 15, 2016, 10:07am UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/5 "2016-01-15T10:07:05Z")
In that case, I have no idea why this is happening. Everything appears to be ticking over nicely, apart from all the duplicate entries. My concern is that this is increasing disk usage, I/O, etc. ---
**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\ **Post date:** [January 15, 2016, 4:11pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/6 "2016-01-15T16:11:39Z")
@magnusbaeck I have a question, but if you have time I wanted to ask your opinion before testing. Is there a difference between these 2? Would this possibly stop my duplication issue? Original: `output { if [type] == "cloudtrail" { elasticsearch { hosts => "MYESCLUSTER"` \<======== **REMOVE HERE** `index => "logstash-cloudtrail" } } else { elasticsearch { hosts => "MYESCLUSTER"` \<======== **REMOVE HERE** `index => "wtf-are-these-logs" } } stdout { codec => "rubydebug" } }` Revised: `output { if [type] == "cloudtrail" { elasticsearch { index => "logstash-cloudtrail" } } else { elasticsearch { index => "wtf-are-these-logs" } } elasticsearch {` \<=============== **ADD HERE** `hosts => "MYESCLUSTER"` \<======== **ADD HERE** `}` \<============================ **ADD HERE** `stdout { codec => "rubydebug" } }` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 15, 2016, 5:20pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/7 "2016-01-15T17:20:09Z")
No, this doesn't make sense. ---
**Author:** ![plonka2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/plonka2000/32/5077_2.png) [@plonka2000](https://discuss.elastic.co/u/plonka2000)\ **Post date:** [January 15, 2016, 5:22pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/8 "2016-01-15T17:22:06Z")
Yeah, sorry, I just literally finished testing it, and it doesn't work. My logic is I'm wondering if the output is registering twice? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 17, 2016, 4:27pm UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/9 "2016-01-17T16:27:32Z")
You can start Logstash with `--debug` to see exactly what configuration Logstash loads. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:15am UTC](https://discuss.elastic.co/t/duplicate-logs-logstash-input-s3/39231/10 "2017-07-06T05:15:26Z")