Load logs from gzip files using codec

Elastic Stack Logstash
1

There is a closed topic regarding the same from 2 years ago ...

But the .rb code has been updated on GitHub.

What is the latest solution to using the gzip-lines codec?

I have installed the codec and am running the following conf ...

input {
        file {
                type => "gzip"
                path => "/home/aptusdb/data/list.txt" #path to list.txt containing path/filename of all .gz files
                start_position => "beginning"
                sincedb_path => "gzip"
                codec => "gzip_lines" #gzip_lines codec
        }
}

output {
        elasticsearch {
                hosts => "localhost"
                index => "trial1"
                document_type => "sold_cars2"
        }
        stdout {}
}

But I am getting the following error ...

[2018-10-05T15:33:45,695][ERROR][filewatch.tailmode.handlers.grow] read_to_eof: general error reading /home/aptusdb/data/list.txt {"error"=>"java.lang.IllegalArgumentException: Object: data/sample/day=2018-09-23/hour=23/sample_data.txt.gz is not a legal argument to this wrapper, cause it doesn't respond to \"read\".", "backtrace"=>["org.jruby.util.IOInputStream.<init>(IOInputStream.java:64)", "org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(JZlibRubyGzipReader.java:115)", "org.jruby.ext.zlib.JZlibRubyGzipReader.initialize19(JZlibRubyGzipReader.java:145)", "org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.call(JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.gen)"]}

2

The file input plugin now has a read mode that may be what you are looking for as it supports reading compressed files.

1 Like
3

Thanks a lot!

Here's a reference for those facing the same issue:

input {
	file {
		#type => "gzip" (not compulsorily required?)
		path => "/home/user/data/**/*" #path to .gz files contained in subfolders
		#sincedb_path => "gzip" (not compulsorily required?)
		mode => "read"
		file_completed_action => "log"
		file_completed_log_path => "/home/user/data/log.txt"
	}
}
1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.