File input for gz files

Elastic Stack Logstash
#1

HI

I have seen few topics on logstash supporting reading from the gz files.

Would like to know if this feature is supported !! I am using logstash 2.1.0. If supported please share me the sample input config.

Below is my config.
input {
file {
type => "gzip"
path => "/syslog/applog/notif/delwin23/sc5/HCPTapp.log.2015_11_29*.gz"
start_position => "beginning"
sincedb_path => "gzip"
codec => "gzip_lines"
}
}

output {

stdout { codec => json }

}

Output:
bash-3.2$ ./logstash -f ns_off.cnf
io/console not supported; tty will not be manipulated
Settings: Default filter workers: 2
Logstash startup completed
Logstash shutdown completed
bash-3.2$

It seems the file is not read.

Thanks

#3

Attaching the debug log in 2 posts - as the max content limit is 7K Chars..and there is no file attachment option

bash-3.2$ ./logstash -f ns_off.cnf --debug
io/console not supported; tty will not be manipulated
Reading config file {:config_file=>"/sysapp/app/elk/logstash-2.1.0/bin/ns_off.cnf", :level=>:debug, :file=>"logstash/agent.rb", :line=>"325", :method=>"local_config"}
Compiled pipeline code:
@inputs = []
@filters = []
@outputs = []
@periodic_flushers = []
@shutdown_flushers = []

      @input_file_1 = plugin("input", "file", LogStash::Util.hash_merge_many({ "type" => ("gzip") }, { "path" => ("/syslog/applog/notif/delwin23/sc5/HCPTapp.log.2015_11_29*.gz") }, { "start_position" => ("beginning") }, { "sincedb_path" => ("gzip") }, { "codec" => ("gzip_lines") }))

      @inputs << @input_file_1

      @output_stdout_2 = plugin("output", "stdout", LogStash::Util.hash_merge_many({ "codec" => ("json") }))

      @outputs << @output_stdout_2

def filter_func(event)
events = [event]
@logger.debug? && @logger.debug("filter received", :event => event.to_hash)
events
end
def output_func(event)
@logger.debug? && @logger.debug("output received", :event => event.to_hash)
@output_stdout_2.handle(event)

end {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"38", :method=>"initialize"}

#4

Hi
I have tried in another OS (RHEL) and the file detection and reading is happening.
However the output is all the junk/special characters which would be in a typical gz files. How to get the output as same in the actual "plain" text format as if in original file. !!

Below is the config
file {
path => "/notif_logs/selfcareApps/selfcare1/HCPTapp.log*.gz"
start_position => "beginning"
codec => "gzip_lines"
}

#5

Did you install the gzip_lines codec?

#6

Hi Mark
Yes. Infact I installed the "Logstash 2.1.0 (All Plugins)" version. Pls check below.

bash-3.2$ find . -name gzip
./vendor/bundle/jruby/1.9/gems/activesupport-4.1.14/lib/active_support/gzip.rb
./vendor/bundle/jruby/1.9/gems/bindata-2.1.0/examples/gzip.rb
./vendor/bundle/jruby/1.9/gems/faraday_middleware-0.10.0/lib/faraday_middleware/gzip.rb
./vendor/bundle/jruby/1.9/gems/gelfd-0.2.0/lib/gelfd/gzip_parser.rb
./vendor/bundle/jruby/1.9/gems/google-api-client-0.8.6/lib/google/api_client/gzip.rb
./vendor/bundle/jruby/1.9/gems/google-api-client-0.8.6/spec/google/api_client/gzip_spec.rb
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip_lines-2.0.2
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip_lines-2.0.2/lib/logstash/codecs/gzip_lines.rb
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip_lines-2.0.2/logstash-codec-gzip_lines.gemspec
./vendor/bundle/jruby/1.9/gems/mimemagic-0.3.0/test/files/application.gzip
./vendor/bundle/jruby/1.9/gems/restforce-2.1.2/lib/restforce/middleware/gzip.rb
./vendor/bundle/jruby/1.9/gems/restforce-2.1.2/spec/unit/middleware/gzip_spec.rb
./vendor/bundle/jruby/1.9/specifications/logstash-codec-gzip_lines-2.0.2.gemspec
bash-3.2$

#7

Hi
Can anyone Pls guide me on this !!

Does logstash supporting reading from gz files and output the "plain" text as in original file.
If so, pls suggest the correct configuration.

Thanks in anticipation.

#8

For the special chars, if the input file is not encoded in UTF-8, you have to set encoding => "YourEncoding" in input plugin.

For the output, do you try with "rubydebug" output, to check that all input/filter chain works fine ?

I mean :
output {
stdout { codec => rubydebug }
}

#9

Hi Thanks for Reply.
As I mentioned in the problem statement - The input file is a gizzed file in gz format. The original file is actually a plain text file (application log4j log file). I have also mentioned my input plugin configuration above.

I don't find any configuration setting "encoding" in the file input plugin.

Pls suggest

#10

OK.
To be clearer, if your gziped input files are written using "ISO-8859-1" charset (or any charset different from UTF-8), use this configuration :

file {
path => "/notif_logs/selfcareApps/selfcare1/HCPTapp.log*.gz"
start_position => "beginning"
codec => gzip_lines { charset => "ISO-8859-1" }
}

#11

wondering why this (obvious) syntax still does not work for me.....

#12

I was still not able to solve this problem. :frowning:

2 Likes
#13

This syntax is not working. You can see this open issue. Also it is know issue for this flavor. So I suppose that currently it is not supported.

The only workaround that I found was the below:

 pipe {
        command => "gunzip --stdout /home/....../test.gz"
      }

Which IHO is not the best.

#14