Decode error with input plugin pipe

Elastic Stack Logstash
1

I tried the following conf
input {
file {
type => "gz_log"
path => ["/data1/logs/logs/20170501/*.log.gz"]
# path => ["/var/log/lastlog"]
start_position => "beginning"
}

pipe {
command => "gunzip /data1/logs/logs/*/*log.gz"
}

}

output {
stdout { codec => rubydebug }
redis {
host => "10.10.0.213"
port => 6379
data_type => "list"
key => "gz_log"
}
}

but it turns out decoding error I think

 "path" => "/data1/logs/logs/20170501/access.log.20170501_331.log.gz",
"@timestamp" => 2017-05-04T09:34:36.876Z,
  "@version" => "1",
      "host" => "superman",
   "message" => "h\\xEA\\x95t\\x80\\xC5\\u0010\\x99\\xCE.I\\xF9R\\xB9I\\t'\\xC0an\\xD4H\\xEC\\xD8Ѥ\\xB3\\xB8\\xB9\\xBC!ED>\\xB5>\\xC6]\\x80G\\xDAv\\xBC[\\xDA,^\\x9B9q3Pq\\xB3\\xBB\\xA3\\xFF\\xA8nf_\\x91q\\x8F\\xABc\\xECՋ\\b`\\x98\\xF7\\x85+Z\\xDE\\a\\x90\\xAF\\xC9\\xFA\\xB3LMK\\x90\\xEB%\\x9BL\\x9D\\e\\xF9\\xE0Y\\xAF\\u001E\\u0015\\xFB\\xED\\xA5\\x98\\\\ƛGk\\xBD\\t\\xB8\\u0001\\xDBR\\x99B\\x91\\xFFW\\xD9v\\x84,\\xAF\\xB9\\x94[qѬ\\xE9\\u0003,\\x94,9\\x9CC\\xBD\\xE4g\\xDE\\xCC\\u0006\\x9E&RE\\u0001C+\\xA5\\x87L*\\xB5\\xEC\\u001D\\xB0\\t\\u0001\\u0013\\xA0\\a;71`F\\u000E\\x93\\u0015\\u0003\\xFB\\xA8\\x81}\\xF7n\\xA2\\x89\\xF1\\xB6]\\x88\\xBB\\xE8\\xF7\\u0017>?\\xA6\\u007F\\xC1;\\xA5\\xCAq.\\xC4Q \\xF8\\xFD\\u0011N\\u001Fb\\xB8?\\xC0g-\\x8B\\xE6\\u0019\\u0000h\\x95֣\\xC5\\xF8rGR\\u0015.ij\\xE6\\xFFԩ\\x90T\\xD5`\\x8B\\xA3\\x8C\\x8E\\u0003\\x97N9r\\xF2\\xCCvp\\u0003\\xBB\\ro^)F\\x96\\x80{\\u0006\\x95\\b\\xE9]\\xC0\\xAA\\u070E\\u00051\\u0015kCg\\x93\\u0017\\e\\x9A\\xDF\\xD9\\xE5[,N\\xA6\\x87Q\\xEF\\xE9\\x997\\xF3\\x8E\\xCE\\xDE\\xF2%\\x81e\\u0011\\u001E4\\xB2\\xBB  '\\x89\\xA6\\x95\\x96X\\xF2\\xACM\\xAD\\xB21\\xF2f\\xE7nnбe\\u0018\\xB7\\xFAh\\xFB\\x9B\\x9B\\xDDM\\x80;ޡq\\x83\\r\\u0001\\x84\\xC7\\xF4/~",
      "type" => "gz_log"

anyone has some idea about this?

2

I think you're misunderstanding how Logstash works. The two inputs you've listed will run independently of each other. The pipe input won't run gunzip what the file input reads.

3

Thank you Magnus, I do this because gzip_lines codec don't support Logstash version>2.0.

What I want is a pipeline:1. input the .gz file 2. unzip the file 3. output to redis

So how can I do this?

4

I don't know if there's a way of doing this except temporarily gunzipping the files outside of Logstash.

5

temprorarily gunzip the files, that is a solution, but it needs a quite a lot of work to verify and monitor.
You mean my setting leads to two inputs parellal working, I know why the file input fail. But why the pipe input also fail? I mean why I can't gunzip by plugin pipe?

6

But why the pipe input also fail?

How do you know it fails?

I mean why I can't gunzip by plugin pipe?

You can, but you won't be able to keep track of much of the files you have processed. If that doesn't matter to you maybe it's an option after all.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.