How to get .gz files using Gzip_lines codec plugin from a pipeline?

Elastic Stack Logstash
1

How to get .gz files using Gzip_lines codec plugin from a pipeline?

input {
  pipeline {
    address => testgz            #has been configured to send .gz files into this pipe
    codec => gzip_lines
  }
}
output {
  stdout { codec => rubydebug }
}

But the message output is still garbled...
Thanks

2

Are you using the correct charset ? I think UTF-8 is default. Maybe you need to set the correct one, if the default doesn't fit.

3

Oh! i just checked and my .gz file is using Binary charset. i will change n try now.

-Update.

[root@linuxclient adsm]# file -i gacs_event_hdlr.log.2019-05-28-03.gz
gacs_event_hdlr.log.2019-05-28-03.gz: application/x-gzip; charset=binary

and setting the charset to "BINARY" does not work. Message output is still garble.

4

The pipeline input ignores codecs (the Event objects are passed in memory without intermediate serialisation); this is a known issue in previous releases of Logstash and will be resolved before pipeline-to-pipeline graduates from beta.

5

Thank you for notifying me of this! I guess I have to do some file manipulation.
Hmm isit actually possible to gunzip files at the filter portion before processing the data?

6

What is the original source? You may be able to use the gzip_lines codec at that point.

7

Actually the source is on the same machine itself at /var/log/adsm/*
Because I have tons of logs there with .log & .out & .log.gz & .out.gz extensions, I replicate the distributor pattern for pipelining the data. Where 1 main pipe will distribute logs with *.log to 1 pipeline and *.out to another pipeline. This is done based on [fields][type] tagging.

Hence this portion is the last pipeline to elasticsearch.

8

What about the gzip plugin working with

input {
  beats {
    port => 5046
    codec => gzip_lines { charset => "BINARY" }
  }
}

Where i ingest .gz files using filebeat on the client side and it doesn't seem to work either.
In any case, i have done it via the manual extraction. But it will be good to know if this works too :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.