Hi all,
It seems there are issues regarding gzip file support, but I can't really find any post that will give me the answer. I have tried to use gzip_lines codec. It does not work.
input {
file {
path => "/path/*.gz"
codec => "gzip_lines"
}
}
output {
stdout { codec => rubydebug }
}
The file input plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html) seems to indicate the "read" mode now supports gzip file processing.
input {
file {
path => "/path/test.log.gz"
mode => "read"
file_completed_action => "log"
file_completed_log_path => "/path/output_completed_log.log"
}
}
output {
stdout { codec => rubydebug }
}
With "mode" set to read, I am getting argument error.
[2019-05-20T09:45:33,403][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-20T09:45:33,449][INFO ][filewatch.observingread ] START, creating Discoverer, Watch with file and sincedb collections
[2019-05-20T09:45:33,454][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-20T09:45:33,665][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::File mode=>"read", path=>["/Users/kwlee/workspace/purge/data/test.log.gz"], discover_interval=>40, file_sort_by=>"path", id=>"18d120c5852688eeea2e022776fe4faaf72378c29c18a692315259782ec16260", file_completed_action=>"log", file_completed_log_path=>"/Users/kwlee/workspace/purge/data/gz-local-read-file-mode-completed.log", stat_interval=>0.1, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b6fdea91-b564-436b-ac87-7020bff4afbb", enable_metric=>true, charset=>"UTF-8">, sincedb_write_interval=>15.0, start_position=>"end", delimiter=>"\n", close_older=>3600.0, sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_direction=>"asc">
Error: wrong number of arguments (given 1, expected 0)
Exception: ArgumentError
Stack: /Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/handlers/read_zip_file.rb:26:in `handle_specifically'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/handlers/base.rb:26:in `handle'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:39:in `read_zip_file'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:102:in `block in process_active'
org/jruby/RubyArray.java:1792:in `each'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:88:in `process_active'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:45:in `process_all_states'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/watch.rb:67:in `iterate_on_state'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/watch.rb:45:in `subscribe'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/observing_read.rb:12:in `subscribe'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/logstash/inputs/file.rb:339:in `run'
/Users/kwlee/workspace/purge/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:297:in `inputworker'
/Users/kwlee/workspace/purge/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:290:in `block in start_input'
Without "mode" set to read, I get unreadable message
"message" => "$\\xED\\x87S\\xF4h\\xC8״\\xA9!E!\\xA7\\x94'\\xE5\\x8C\\u0005m\\x99\\x8Cs\\u007F\\xCCU\\u0006\\x86H\\u001AS\\xFB\\xEE\\u0018\\x96\\u0006\\x9C\\u0002\\x96\\x82~M4(\\xC0\\x91`\\u001A\\x9E\\xA5\\u0000\\xB0\\xCB|\\x96&\\xC14\\xA2XJ\\x82\\xD3\\xE0=8\\r\\xB5\\xEE\\xC9$\\b\\xA7\\xE1\\xC5!L\\xA4\\xDB\\xE8dȳӨ\\xB6\\u00038\\r\\xB3\\u0016\\\\\\x83\\u0006\\u000FZ\\xAF9\\u0005N\\xC33'\\x97\\xD3eQ8\\x8D\\x802\\xDBAf|{\\t/&\\xE4\\xB2Q\\xDA/\\xE9m\\xF3$C\\u001E\\xF5X\\u0018\\x94\\x9A\\u0010\\u0016\\x86\\xE7_Ǻ\\u000EN\\u0016@\\b\\v\\x83+\\xB3\\xA9\\xEB^H\\xC2VԬ\\xC0\\x8E\\xDBn%ۀ\\xFD'B\\xE9\\xF5\\x9E\\u0001\\xB3\\u0018\\u0003\\x8B\\x875\\r\\xA1x09\\x99\\xF8g\\rk\\u001A\\\"\\xF0\\x9E\\u001D9\\xD9I\\u0003\\u0003\\x8B\\xEA\\x83zBX\\xD3\\xC1\\u0012\\u0004Cc3YO\\xC7\\xC2\\b\\xEF\\xB4\\u0018\\u000E5\\x9Es\\x94bK&aa\\xE4\\xE0\\xCER\\xB4\\xF3\\xA0\\u0012\\b\\u0018\\xB1\\xA2\\xF9",`
I really appreciate If someone can point me in the right direction.
Thanks