# File gzip output creating wrong file?

**URL:** <https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350>\
**Category:** Logstash\
**Created:** [May 27, 2015, 12:09am UTC](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350 "2015-05-27T00:09:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\
**Post date:** [May 27, 2015, 12:10am UTC](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350/1 "2015-05-27T00:10:00Z")

</div>

Sorry about this - I managed to delete my previous post somehow!

My issue is that I have an gzip output like:  
file {  
path =\> "/var/data/logstash/archive-%{+YYYY-MM-dd}.log.gz"  
gzip =\> true  
}  
The files created by that are transferred to S3 every day just after midnight. The issue is that the day _after_ an archive is created and transferred, it's getting made again with a single log from the previous day. So for example, I'm getting a gzip file named 'archive-2015-05-22.log.gz', containing a single log with timestamp '2015-05-22T07:43:34.785Z' and received\_at '2015-05-22 07:43:34 UTC' being created at approximately 2015-05-23 07:43:34 UTC (Note the changed day). When that new file is uploaded, it replaces the proper archive for that day.

In the previous post, someone replied suggesting I use the S3 output instead, which I will investigate, although I see know option (yet) to compress the output.

The logstash config is:

input {  
lumberjack {  
port =\> 6782  
ssl\_certificate =\> "..."  
ssl\_key =\> "..."  
}  
}

filter {

if [type] == "tomcat" {  
grok {  
match =\> ["message", "%{TIMESTAMP\_ISO8601:time} | %{GREEDYDATA:appname} | [%{GREEDYDATA:thread}] | %{LOGLEVEL:loglevel} ? | %{GREEDYDATA:class} | [%{GREEDYDATA:mdc}] | [%{GREEDYDATA:msg}]" ]  
match =\> ["message", "%{TIMESTAMP\_ISO8601:time} | %{GREEDYDATA:appname} | [%{GREEDYDATA:thread}] | %{LOGLEVEL:loglevel} ? | %{GREEDYDATA:class} | %{GREEDYDATA:msg}" ]  
match =\> ["message", "%{TIMESTAMP\_ISO8601:time} | %{GREEDYDATA:appname} | [%{GREEDYDATA:thread}] | %{LOGLEVEL:loglevel} ? | %{GREEDYDATA:class} | [%{GREEDYDATA:msg}]" ]  
match =\> ["message", "%{GREEDYDATA:msg}"]  
add\_field =\> ["facility", "tomcat"]  
}

```
multiline {
  # If it doesnt start with a date join it to the previous line
  pattern => "\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2},\d{3}\+\d{4} .*"  
  negate => "true"
  what => "previous"
}

kv {
  source => "mdc"
  field_split => ", "
}

```

}

mutate {  
uppercase =\> ["loglevel"]  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_tag =\> ["mutated"]  
}

date {  
match =\> ["time", "ISO8601", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd'T'HH:mm:ss,SSS", "MMM dd YYY HH:mm:ss" , "MMM d YYY HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss Z", "YYYY/MM/dd HH:mm:ss"]  
target =\> "@timestamp"  
add\_tag =\> ["date\_changed"]  
}  
}

output {

file {  
path =\> "/var/data/logstash/archive-%{+YYYY-MM-dd}.log.gz"  
gzip =\> true  
}

elasticsearch {  
cluster =\> "..."  
node\_name =\> "..."  
host =\> "127.0.0.1"  
port =\> "9300"  
protocol =\> "transport"  
}

}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 27, 2015, 12:12am UTC](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350/2 "2015-05-27T00:12:20Z")

</div>

Your last post is here - [Logstash file output creating wrong file?](https://discuss.elastic.co/t/logstash-file-output-creating-wrong-file/1349)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 27, 2015, 12:12am UTC](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350/3 "2015-05-27T00:12:24Z")

</div>