# Decode error with input plugin pipe

**URL:** <https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525>\
**Category:** Logstash\
**Created:** [May 4, 2017, 9:40am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525 "2017-05-04T09:40:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\
**Post date:** [May 4, 2017, 9:40am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/1 "2017-05-04T09:40:02Z")

</div>

I tried the following conf  
input {  
file {  
type =\> "gz\_log"  
path =\> ["/data1/logs/logs/20170501/\*.log.gz"]  
# path =\> ["/var/log/lastlog"]  
start\_position =\> "beginning"  
}

pipe {  
command =\> "gunzip /data1/logs/logs/\*/\*log.gz"  
}

}

output {  
stdout { codec =\> rubydebug }  
redis {  
host =\> "10.10.0.213"  
port =\> 6379  
data\_type =\> "list"  
key =\> "gz\_log"  
}  
}

but it turns out decoding error I think

```
 "path" => "/data1/logs/logs/20170501/access.log.20170501_331.log.gz",
"@timestamp" => 2017-05-04T09:34:36.876Z,
  "@version" => "1",
      "host" => "superman",
   "message" => "h\\xEA\\x95t\\x80\\xC5\\u0010\\x99\\xCE.I\\xF9R\\xB9I\\t'\\xC0an\\xD4H\\xEC\\xD8Ѥ\\xB3\\xB8\\xB9\\xBC!ED>\\xB5>\\xC6]\\x80G\\xDAv\\xBC[\\xDA,^\\x9B9q3Pq\\xB3\\xBB\\xA3\\xFF\\xA8nf_\\x91q\\x8F\\xABc\\xECՋ\\b`\\x98\\xF7\\x85+Z\\xDE\\a\\x90\\xAF\\xC9\\xFA\\xB3LMK\\x90\\xEB%\\x9BL\\x9D\\e\\xF9\\xE0Y\\xAF\\u001E\\u0015\\xFB\\xED\\xA5\\x98\\\\ƛGk\\xBD\\t\\xB8\\u0001\\xDBR\\x99B\\x91\\xFFW\\xD9v\\x84,\\xAF\\xB9\\x94[qѬ\\xE9\\u0003,\\x94,9\\x9CC\\xBD\\xE4g\\xDE\\xCC\\u0006\\x9E&RE\\u0001C+\\xA5\\x87L*\\xB5\\xEC\\u001D\\xB0\\t\\u0001\\u0013\\xA0\\a;71`F\\u000E\\x93\\u0015\\u0003\\xFB\\xA8\\x81}\\xF7n\\xA2\\x89\\xF1\\xB6]\\x88\\xBB\\xE8\\xF7\\u0017>?\\xA6\\u007F\\xC1;\\xA5\\xCAq.\\xC4Q \\xF8\\xFD\\u0011N\\u001Fb\\xB8?\\xC0g-\\x8B\\xE6\\u0019\\u0000h\\x95֣\\xC5\\xF8rGR\\u0015.ĳ\\xE6\\xFFԩ\\x90T\\xD5`\\x8B\\xA3\\x8C\\x8E\\u0003\\x97N9r\\xF2\\xCCvp\\u0003\\xBB\\ro^)F\\x96\\x80{\\u0006\\x95\\b\\xE9]\\xC0\\xAA\\u070E\\u00051\\u0015kCg\\x93\\u0017\\e\\x9A\\xDF\\xD9\\xE5[,N\\xA6\\x87Q\\xEF\\xE9\\x997\\xF3\\x8E\\xCE\\xDE\\xF2%\\x81e\\u0011\\u001E4\\xB2\\xBB '\\x89\\xA6\\x95\\x96X\\xF2\\xACM\\xAD\\xB21\\xF2f\\xE7nnбe\\u0018\\xB7\\xFAh\\xFB\\x9B\\x9B\\xDDM\\x80;ޡq\\x83\\r\\u0001\\x84\\xC7\\xF4/~",
      "type" => "gz_log"

```

anyone has some idea about this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 4, 2017, 9:54am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/2 "2017-05-04T09:54:37Z")

</div>

I think you're misunderstanding how Logstash works. The two inputs you've listed will run independently of each other. The pipe input won't run gunzip what the file input reads.

---

<div class="post-metadata">

**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\
**Post date:** [May 4, 2017, 9:58am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/3 "2017-05-04T09:58:24Z")

</div>

Thank you Magnus, I do this because gzip\_lines codec don't support Logstash version\>2.0.

What I want is a pipeline:1. input the .gz file 2. unzip the file 3. output to redis

So how can I do this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 4, 2017, 10:07am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/4 "2017-05-04T10:07:17Z")

</div>

I don't know if there's a way of doing this except temporarily gunzipping the files outside of Logstash.

---

<div class="post-metadata">

**Author:** ![LmYjQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmyjq/32/97480_2.png) [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\
**Post date:** [May 4, 2017, 10:29am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/5 "2017-05-04T10:29:06Z")

</div>

temprorarily gunzip the files, that is a solution, but it needs a quite a lot of work to verify and monitor.  
You mean my setting leads to two inputs parellal working, I know why the file input fail. But why the pipe input also fail? I mean why I can't gunzip by plugin pipe?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 4, 2017, 11:48am UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/6 "2017-05-04T11:48:16Z")

</div>

> But why the pipe input also fail?

How do you know it fails?

> I mean why I can't gunzip by plugin pipe?

You can, but you won't be able to keep track of much of the files you have processed. If that doesn't matter to you maybe it's an option after all.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2017, 12:02pm UTC](https://discuss.elastic.co/t/decode-error-with-input-plugin-pipe/84525/7 "2017-06-01T12:02:37Z")

</div>

Closed on June 1, 2017, 12:02pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
