# Logstash not parsing every file

**URL:** <https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372>\
**Category:** Logstash\
**Created:** [May 23, 2019, 7:53am UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372 "2019-05-23T07:53:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\
**Post date:** [May 23, 2019, 7:53am UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/1 "2019-05-23T07:53:29Z")

</div>

Hi All.

I have logstash set up to parse/ingest a local directory on my server.

The directory is for a CDN log files which are fetched every 10 minutes.  
Unfortuantly the CDN produces a new log file multiple times a minutes ( so for example I have over 1000 log files for 1am-8am today).

The logs are never appended too, so just need to be read and then forgotten about.

Logstash seems to be struggling to parse these.

if i go into the directoy and run "zcat \*.log.gz | wc -l it shows 103,884 lines, yet only 11,620 hits are showing in Kibana for today.

I would expect kibana to show 103,884 lines.

Looking in the file\_completed\_log it does seem to be missing quite a few out.

My config file for input is below -

input {  
file {  
path =\> "/data/logs/\*.log.gz"  
sincedb\_path =\> "/data/logstash-db/sincedb"  
mode =\> "read"  
file\_completed\_action =\> "log"  
file\_completed\_log\_path =\> "/data/logstash-db/file\_completed\_log"  
}  
}

The log files are called - cds\_20190522-154421-57378698007ch4.log.gz (ect ect)

Can anyone think of why this could be happening? Is logstash known for struggling with lots of small files?

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [May 27, 2019, 8:55pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/2 "2019-05-27T20:55:02Z")

</div>

Have you tried adding the gzip codec to your file input?  
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip\_lines.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip_lines.html)

---

<div class="post-metadata">

**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\
**Post date:** [May 28, 2019, 6:36am UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/3 "2019-05-28T06:36:38Z")

</div>

Hi,

Thanks for your response!.

This plugin is all ready installed.

The logstash works for some time, it has been working over the weekend but then randomly decided to stop @ 6pm on sunday (with no errors in the logs).

It's really frustrating me how it works for a few days, then just crashes/stops for no reason.

I then have to restart elasticsearch/logstash and it starts working again.

---

<div class="post-metadata">

**Author:** ![kirangavali](https://avatars.discourse-cdn.com/v4/letter/k/77aa72/32.png) [@kirangavali](https://discuss.elastic.co/u/kirangavali)\
**Post date:** [May 28, 2019, 10:29am UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/4 "2019-05-28T10:29:59Z")

</div>

Hello @jamesp220291 can you please share the output plugin configuration details.  
I mean how you configure the logstash output with the elasticsearch as input and logstash output with kibana as input.

---

<div class="post-metadata">

**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\
**Post date:** [May 28, 2019, 12:32pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/5 "2019-05-28T12:32:15Z")

</div>

Hi

See below -

input {  
file {  
path =\> "/data/logs/\*.log.gz"  
sincedb\_path =\> "/data/logstash-db/sincedb"  
mode =\> "read"  
file\_completed\_action =\> "log"  
file\_completed\_log\_path =\> "/data/logstash-db/file\_completed\_log"  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "logs-%{+YYYY.MM.dd}"  
document\_type =\> "logs"  
}

# stdout { codec =\> rubydebug }

}

This is the input/output part of the logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2019, 12:32pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-every-file/182372/6 "2019-06-25T12:32:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.