# File input for gz files

**URL:** <https://discuss.elastic.co/t/file-input-for-gz-files/35822>\
**Category:** Logstash\
**Created:** [November 29, 2015, 9:06am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822 "2015-11-29T09:06:22Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [November 29, 2015, 9:06am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/1 "2015-11-29T09:06:22Z")

</div>

HI

I have seen few topics on logstash supporting reading from the gz files.

Would like to know if this feature is supported !! I am using logstash 2.1.0. If supported please share me the sample input config.

Below is my config.  
input {  
file {  
type =\> "gzip"  
path =\> "/syslog/applog/notif/delwin23/sc5/HCPTapp.log.2015\_11\_29\*.gz"  
start\_position =\> "beginning"  
sincedb\_path =\> "gzip"  
codec =\> "gzip\_lines"  
}  
}

output {

```
stdout { codec => json }

```

}

Output:  
bash-3.2$ ./logstash -f ns\_off.cnf  
io/console not supported; tty will not be manipulated  
Settings: Default filter workers: 2  
Logstash startup completed  
Logstash shutdown completed  
bash-3.2$

It seems the file is not read.

Thanks

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [November 29, 2015, 10:25am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/3 "2015-11-29T10:25:26Z")

</div>

Attaching the debug log in 2 posts - as the max content limit is 7K Chars..and there is no file attachment option

bash-3.2$ ./logstash -f ns\_off.cnf --debug  
io/console not supported; tty will not be manipulated  
Reading config file {:config\_file=\>"/sysapp/app/elk/logstash-2.1.0/bin/ns\_off.cnf", :level=\>:debug, :file=\>"logstash/agent.rb", :line=\>"325", :method=\>"local\_config"}  
Compiled pipeline code:  
@inputs = []  
@filters = []  
@outputs = []  
@periodic\_flushers = []  
@shutdown\_flushers = []

```
      @input_file_1 = plugin("input", "file", LogStash::Util.hash_merge_many({ "type" => ("gzip") }, { "path" => ("/syslog/applog/notif/delwin23/sc5/HCPTapp.log.2015_11_29*.gz") }, { "start_position" => ("beginning") }, { "sincedb_path" => ("gzip") }, { "codec" => ("gzip_lines") }))

      @inputs << @input_file_1

      @output_stdout_2 = plugin("output", "stdout", LogStash::Util.hash_merge_many({ "codec" => ("json") }))

      @outputs << @output_stdout_2

```

def filter\_func(event)  
events = [event]  
@logger.debug? && @logger.debug("filter received", :event =\> event.to\_hash)  
events  
end  
def output\_func(event)  
@logger.debug? && @logger.debug("output received", :event =\> event.to\_hash)  
@output\_stdout\_2.handle(event)

end {:level=\>:debug, :file=\>"logstash/pipeline.rb", :line=\>"38", :method=\>"initialize"}

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [November 29, 2015, 5:01pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/4 "2015-11-29T17:01:38Z")

</div>

Hi  
I have tried in another OS (RHEL) and the file detection and reading is happening.  
However the output is all the junk/special characters which would be in a typical gz files. How to get the output as same in the actual "plain" text format as if in original file. !!

Below is the config  
file {  
path =\> "/notif\_logs/selfcareApps/selfcare1/HCPTapp.log\*.gz"  
start\_position =\> "beginning"  
codec =\> "gzip\_lines"  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 29, 2015, 9:08pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/5 "2015-11-29T21:08:51Z")

</div>

Did you install the `gzip_lines` codec?

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [November 29, 2015, 11:23pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/6 "2015-11-29T23:23:53Z")

</div>

Hi Mark  
Yes. Infact I installed the "Logstash 2.1.0 (All Plugins)" version. Pls check below.

bash-3.2$ find . -name _gzip_  
./vendor/bundle/jruby/1.9/gems/activesupport-4.1.14/lib/active\_support/gzip.rb  
./vendor/bundle/jruby/1.9/gems/bindata-2.1.0/examples/gzip.rb  
./vendor/bundle/jruby/1.9/gems/faraday\_middleware-0.10.0/lib/faraday\_middleware/gzip.rb  
./vendor/bundle/jruby/1.9/gems/gelfd-0.2.0/lib/gelfd/gzip\_parser.rb  
./vendor/bundle/jruby/1.9/gems/google-api-client-0.8.6/lib/google/api\_client/gzip.rb  
./vendor/bundle/jruby/1.9/gems/google-api-client-0.8.6/spec/google/api\_client/gzip\_spec.rb  
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip\_lines-2.0.2  
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip\_lines-2.0.2/lib/logstash/codecs/gzip\_lines.rb  
./vendor/bundle/jruby/1.9/gems/logstash-codec-gzip\_lines-2.0.2/logstash-codec-gzip\_lines.gemspec  
./vendor/bundle/jruby/1.9/gems/mimemagic-0.3.0/test/files/application.gzip  
./vendor/bundle/jruby/1.9/gems/restforce-2.1.2/lib/restforce/middleware/gzip.rb  
./vendor/bundle/jruby/1.9/gems/restforce-2.1.2/spec/unit/middleware/gzip\_spec.rb  
./vendor/bundle/jruby/1.9/specifications/logstash-codec-gzip\_lines-2.0.2.gemspec  
bash-3.2$

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [December 1, 2015, 8:49am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/7 "2015-12-01T08:49:48Z")

</div>

Hi  
Can anyone Pls guide me on this !!

Does logstash supporting reading from gz files and output the "plain" text as in original file.  
If so, pls suggest the correct configuration.

Thanks in anticipation.

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [December 6, 2015, 10:07pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/8 "2015-12-06T22:07:42Z")

</div>

For the special chars, if the input file is not encoded in UTF-8, you have to set encoding =\> "YourEncoding" in input plugin.

For the output, do you try with "rubydebug" output, to check that all input/filter chain works fine ?

I mean :  
output {  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [December 7, 2015, 3:51am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/9 "2015-12-07T03:51:40Z")

</div>

Hi Thanks for Reply.  
As I mentioned in the problem statement - The input file is a gizzed file in gz format. The original file is actually a plain text file (application log4j log file). I have also mentioned my input plugin configuration above.

I don't find any configuration setting "encoding" in the file input plugin.

Pls suggest

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [December 7, 2015, 8:58am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/10 "2015-12-07T08:58:45Z")

</div>

OK.  
To be clearer, if your gziped input files are written using "ISO-8859-1" charset (or any charset different from UTF-8), use this configuration :

file {  
path =\> "/notif\_logs/selfcareApps/selfcare1/HCPTapp.log\*.gz"  
start\_position =\> "beginning"  
codec =\> gzip\_lines { charset =\> "ISO-8859-1" }  
}

---

<div class="post-metadata">

**Author:** ![Diz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diz/32/7164_2.png) [@Diz](https://discuss.elastic.co/u/Diz)\
**Post date:** [January 14, 2016, 3:51pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/11 "2016-01-14T15:51:54Z")

</div>

wondering why this (obvious) syntax still does not work for me.....

---

<div class="post-metadata">

**Author:** ![mvenkat\_in](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\
**Post date:** [January 29, 2016, 3:40am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/12 "2016-01-29T03:40:36Z")

</div>

I was still not able to solve this problem. ☹

---

<div class="post-metadata">

**Author:** ![lmpampaletakis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmpampaletakis/32/13455_2.png) [@lmpampaletakis](https://discuss.elastic.co/u/lmpampaletakis)\
**Post date:** [November 28, 2016, 1:36pm UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/13 "2016-11-28T13:36:30Z")

</div>

This syntax is not working. You can see [this](https://github.com/elastic/logstash/issues/1817) open issue. Also it is know issue for this [flavor](https://github.com/logstash-plugins/logstash-codec-gzip_lines/issues/4). So I suppose that currently it is not supported.

The only workaround that I found was the below:

```
 pipe {
        command => "gunzip --stdout /home/....../test.gz"
      }

```

Which IHO is not the best.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/file-input-for-gz-files/35822/14 "2017-07-06T04:30:10Z")

</div>