How to get .gz files using Gzip_lines codec plugin from a pipeline?

How to get .gz files using Gzip_lines codec plugin from a pipeline?

input {
  pipeline {
    address => testgz            #has been configured to send .gz files into this pipe
    codec => gzip_lines
output {
  stdout { codec => rubydebug }

But the message output is still garbled...

Are you using the correct charset ? I think UTF-8 is default. Maybe you need to set the correct one, if the default doesn't fit.

Oh! i just checked and my .gz file is using Binary charset. i will change n try now.


[root@linuxclient adsm]# file -i gacs_event_hdlr.log.2019-05-28-03.gz
gacs_event_hdlr.log.2019-05-28-03.gz: application/x-gzip; charset=binary

and setting the charset to "BINARY" does not work. Message output is still garble.

The pipeline input ignores codecs (the Event objects are passed in memory without intermediate serialisation); this is a known issue in previous releases of Logstash and will be resolved before pipeline-to-pipeline graduates from beta.

Thank you for notifying me of this! I guess I have to do some file manipulation.
Hmm isit actually possible to gunzip files at the filter portion before processing the data?

What is the original source? You may be able to use the gzip_lines codec at that point.

Actually the source is on the same machine itself at /var/log/adsm/*
Because I have tons of logs there with .log & .out & .log.gz & .out.gz extensions, I replicate the distributor pattern for pipelining the data. Where 1 main pipe will distribute logs with *.log to 1 pipeline and *.out to another pipeline. This is done based on [fields][type] tagging.

Hence this portion is the last pipeline to elasticsearch.

What about the gzip plugin working with

input {
  beats {
    port => 5046
    codec => gzip_lines { charset => "BINARY" }

Where i ingest .gz files using filebeat on the client side and it doesn't seem to work either.
In any case, i have done it via the manual extraction. But it will be good to know if this works too :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.