Does Logstash (7.0.1) support gzip file input?

Elastic Stack Logstash
1

Hi all,

It seems there are issues regarding gzip file support, but I can't really find any post that will give me the answer. I have tried to use gzip_lines codec. It does not work.

input {
    file {
        path => "/path/*.gz"
        codec => "gzip_lines"
    }
}

output {
    stdout { codec => rubydebug }
}

The file input plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html) seems to indicate the "read" mode now supports gzip file processing.

input {
    file {
        path => "/path/test.log.gz" 
        mode => "read"
        file_completed_action => "log"
        file_completed_log_path => "/path/output_completed_log.log"
    }
}

output {
  stdout { codec => rubydebug }
}

With "mode" set to read, I am getting argument error.

[2019-05-20T09:45:33,403][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-20T09:45:33,449][INFO ][filewatch.observingread  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-05-20T09:45:33,454][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-20T09:45:33,665][ERROR][logstash.javapipeline    ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File mode=>"read", path=>["/Users/kwlee/workspace/purge/data/test.log.gz"], discover_interval=>40, file_sort_by=>"path", id=>"18d120c5852688eeea2e022776fe4faaf72378c29c18a692315259782ec16260", file_completed_action=>"log", file_completed_log_path=>"/Users/kwlee/workspace/purge/data/gz-local-read-file-mode-completed.log", stat_interval=>0.1, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b6fdea91-b564-436b-ac87-7020bff4afbb", enable_metric=>true, charset=>"UTF-8">, sincedb_write_interval=>15.0, start_position=>"end", delimiter=>"\n", close_older=>3600.0, sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_direction=>"asc">
  Error: wrong number of arguments (given 1, expected 0)
  Exception: ArgumentError
  Stack: /Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/handlers/read_zip_file.rb:26:in `handle_specifically'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/handlers/base.rb:26:in `handle'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:39:in `read_zip_file'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:102:in `block in process_active'
org/jruby/RubyArray.java:1792:in `each'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:88:in `process_active'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/read_mode/processor.rb:45:in `process_all_states'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/watch.rb:67:in `iterate_on_state'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/watch.rb:45:in `subscribe'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/observing_read.rb:12:in `subscribe'
/Users/kwlee/workspace/purge/logstash-7.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/logstash/inputs/file.rb:339:in `run'
/Users/kwlee/workspace/purge/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:297:in `inputworker'
/Users/kwlee/workspace/purge/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:290:in `block in start_input'

Without "mode" set to read, I get unreadable message

   "message" => "$\\xED\\x87S\\xF4h\\xC8״\\xA9!E!\\xA7\\x94'\\xE5\\x8C\\u0005m\\x99\\x8Cs\\u007F\\xCCU\\u0006\\x86H\\u001AS\\xFB\\xEE\\u0018\\x96\\u0006\\x9C\\u0002\\x96\\x82~M4(\\xC0\\x91`\\u001A\\x9E\\xA5\\u0000\\xB0\\xCB|\\x96&\\xC14\\xA2XJ\\x82\\xD3\\xE0=8\\r\\xB5\\xEE\\xC9$\\b\\xA7\\xE1\\xC5!L\\xA4\\xDB\\xE8dȳӨ\\xB6\\u00038\\r\\xB3\\u0016\\\\\\x83\\u0006\\u000FZ\\xAF9\\u0005N\\xC33'\\x97\\xD3eQ8\\x8D\\x802\\xDBAf|{\\t/&\\xE4\\xB2Q\\xDA/\\xE9m\\xF3$C\\u001E\\xF5X\\u0018\\x94\\x9A\\u0010\\u0016\\x86\\xE7_Ǻ\\u000EN\\u0016@\\b\\v\\x83+\\xB3\\xA9\\xEB^H\\xC2VԬ\\xC0\\x8E\\xDBn%ۀ\\xFD'B\\xE9\\xF5\\x9E\\u0001\\xB3\\u0018\\u0003\\x8B\\x875\\r\\xA1x09\\x99\\xF8g\\rk\\u001A\\\"\\xF0\\x9E\\u001D9\\xD9I\\u0003\\u0003\\x8B\\xEA\\x83zBX\\xD3\\xC1\\u0012\\u0004Cc3YO\\xC7\\xC2\\b\\xEF\\xB4\\u0018\\u000E5\\x9Es\\x94bK&aa\\xE4\\xE0\\xCER\\xB4\\xF3\\xA0\\u0012\\b\\u0018\\xB1\\xA2\\xF9",`

I really appreciate If someone can point me in the right direction.

Thanks

2

That's odd. The line that is throwing an exception is this. The readline method of a java.io.BufferedReader does not take an argument, so I would expect that exception. However, it works just fine for me.

3

Thanks for the response.

Your logstash config file is somewhat similar to following, and it works for you. Is that right?

input {
    file {
        path => "/path/test.log.gz" 
        mode => "read"
        file_completed_action => "log"
        file_completed_log_path => "/path/output_completed_log.log"
    }
}

output {
  stdout { codec => rubydebug }
}

I download the logstash (tar.gz file) from https://www.elastic.co/downloads/logstash, and I am currently running the logstash on mac.

# ruby --version
ruby 2.3.7p456 (2018-03-28 revision 63024) [universal.x86_64-darwin18]

# java --version
openjdk 12.0.1 2019-04-16
OpenJDK Runtime Environment (build 12.0.1+12)
OpenJDK 64-Bit Server VM (build 12.0.1+12, mixed mode, sharing)

Do you have similar setup environment?

Thanks

4

Correct. The paths are different but the setup is the same.

jruby 9.2.7.0 (2.5.3)
OpenJDK 64-Bit Server VM 25.201-b09 on 1.8.0_201-b09

5

The java version on my mac could be the cause of the issue. I try the same lagstash config file on the Linux machine with java 8, and it works.

$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

I did not realize this could be an issue since everything else worked. I will try to install java 8 on my mac to see if it fixes the issue.

Thanks again for your help.

6

Hi

I have installed java 8 on my mac, and it seems it is working.
I am just wondering if I should open a ticket to track this?

Thanks
Kc

7

I think so, although Java 12 is not on the support matrix for logstash.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.