How do we get logstash to process a gzip file as input?

I have trouble getting logstash to process gzip file . i have installed the gzip codec and referenced it in the code as well . During execution , logstash says " Error: not in gzip format" even the file is gzip . This execution is on a mac . Am i missing anything ?

Here's the code : -

input {
file {
type => "gzip"
path => "/Users/sn/Desktop/directories/*.gz"
mode => "read"
#stdin {}
start_position => "beginning"
codec => gzip_lines
sincedb_path => "/tmp/gzip.db"
}
}

The resulting logstash execution :slight_smile:

[2021-04-22T13:28:47,644][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-04-22T13:29:17,738][ERROR][logstash.javapipeline    ][main][e8c82436c92146e621c5467102eb49d914b26d1498d17969725c60c02b386f3c] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File mode=>"read", start_position=>"beginning", path=>["/Users/sn/Desktop/directories/*.gz"], codec=><LogStash::Codecs::GzipLines id=>"gzip_lines_16daf970-c444-45e8-ace1-1cb6cb67e265", enable_metric=>true, charset=>"UTF-8">, id=>"e8c82436c92146e621c5467102eb49d914b26d1498d17969725c60c02b386f3c", type=>"gzip", sincedb_path=>"/tmp/gzip.db", enable_metric=>true, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>
  Error: not in gzip format
  Exception: Zlib::GzipFile::Error
  Stack: org/jruby/ext/zlib/JZlibRubyGzipReader.java:148:in `initialize'
org/jruby/ext/zlib/JZlibRubyGzipReader.java:92:in `new'
org/jruby/ext/zlib/JZlibRubyGzipReader.java:83:in `new'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-codec-gzip_lines-3.0.4/lib/logstash/codecs/gzip_lines.rb:35:in `decode'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/logstash/inputs/file/patch.rb:6:in `accept'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in `method_missing'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-codec-multiline-3.0.10/lib/logstash/codecs/identity_map_codec.rb:178:in `accept'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/logstash/inputs/file_listener.rb:39:in `accept'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/handlers/read_zip_file.rb:33:in `handle_specifically'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/handlers/base.rb:26:in `handle'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/processor.rb:25:in `read_zip_file'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/processor.rb:88:in `block in process_active'
org/jruby/RubyArray.java:1809:in `each'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/processor.rb:73:in `process_active'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/read_mode/processor.rb:31:in `process_all_states'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/watch.rb:70:in `iterate_on_state'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/watch.rb:44:in `subscribe'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/filewatch/observing_read.rb:12:in `subscribe'
/usr/local/Cellar/logstash-full/7.11.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.3/lib/logstash/inputs/file.rb:364:in `run'
/usr/local/Cellar/logstash-full/7.11.2/libexec/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/local/Cellar/logstash-full/7.11.2/libexec/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'
[2021-04-22T13:29:18,764][INFO ][filewatch.observingread  ][main][e8c82436c92146e621c5467102eb49d914b26d1498d17969725c60c02b386f3c] QUIT - closing all files and shutting down.

Update:-

Got the gzip file to work without the gzip_codec . just the usual input and mode => "read"

path => "/Users/sn/Desktop/directories/*.gz"
mode => "read"
file_completed_action => "log"
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.