Logstash not parsing every file

[jamesp220291 (james)]

Hi All.

I have logstash set up to parse/ingest a local directory on my server.

The directory is for a CDN log files which are fetched every 10 minutes.
Unfortuantly the CDN produces a new log file multiple times a minutes ( so for example I have over 1000 log files for 1am-8am today).

The logs are never appended too, so just need to be read and then forgotten about.

Logstash seems to be struggling to parse these.

if i go into the directoy and run "zcat *.log.gz | wc -l it shows 103,884 lines, yet only 11,620 hits are showing in Kibana for today.

I would expect kibana to show 103,884 lines.

Looking in the file_completed_log it does seem to be missing quite a few out.

My config file for input is below -

input {
file {
path => "/data/logs/*.log.gz"
sincedb_path => "/data/logstash-db/sincedb"
mode => "read"
file_completed_action => "log"
file_completed_log_path => "/data/logstash-db/file_completed_log"
}
}

The log files are called - cds_20190522-154421-57378698007ch4.log.gz (ect ect)

Can anyone think of why this could be happening? Is logstash known for struggling with lots of small files?

[AquaX (Andreas Helmer)]

Have you tried adding the gzip codec to your file input?
https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip_lines.html

[jamesp220291 (james)]

Hi,

Thanks for your response!.

This plugin is all ready installed.

The logstash works for some time, it has been working over the weekend but then randomly decided to stop @ 6pm on sunday (with no errors in the logs).

It's really frustrating me how it works for a few days, then just crashes/stops for no reason.

I then have to restart elasticsearch/logstash and it starts working again.

[kirangavali (Kiran)]

Hello @jamesp220291 can you please share the output plugin configuration details.
I mean how you configure the logstash output with the elasticsearch as input and logstash output with kibana as input.

[jamesp220291 (james)]

Hi

See below -

input {
file {
path => "/data/logs/*.log.gz"
sincedb_path => "/data/logstash-db/sincedb"
mode => "read"
file_completed_action => "log"
file_completed_log_path => "/data/logstash-db/file_completed_log"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{+YYYY.MM.dd}"
document_type => "logs"
}

stdout { codec => rubydebug }

}

This is the input/output part of the logs.