Ingesting .gz files in elastic

[hsalim]

Hello,
I know this topic has been discussed in the past, but I am unable to find any new information. So asking again to see if anyone can offer an update.

Is there a way to ingest .gz files in elastic without unzipping first?

Thanks,
Hamid

[warkolm (Mark Walkom)]

You will still need to decompress them

[hsalim]

Thanks.

Then what does 'read' mode in file input plugin accomplish?

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_read_mode

[warkolm (Mark Walkom)]

In this mode the plugin treats each file as if it is content complete, that is, a finite stream of lines and now EOF is significant.

Basically it means that it'll treat the file as if there will be no new data coming into it.

[hsalim]

So best to unzip prior to ingestion by logstash seems to be the right approach?

[warkolm (Mark Walkom)]

You can use this to read the zipped files in - https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip_lines.html

[hsalim]

Thank you. not sure how to explain why the following works then?

input {
file {
id => "xyz"
path => "/opt/dir1/dir2/local/*.gz"
mode => "read"
sincedb_path => "/dev/null"
}
}