# How to efficiently parse log files in logstash?

**URL:** <https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361>\
**Category:** Logstash\
**Created:** [June 11, 2018, 12:24pm UTC](https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361 "2018-06-11T12:24:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 11, 2018, 12:24pm UTC](https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361/1 "2018-06-11T12:24:17Z")

</div>

Hi,

I have to parse huge amount of log files in logstash. I stored all log files in one directory with .gz format. when I try to extract those logs the size becomes very high. Let's say the log file size is 50MB when it is in .gz format. but after extract that it will become 1 GB. can I parse these logs in effective way instead of unzip all the log files at a time? or can I extract those log files one by one when once a log files is completely parsed in logstash?

Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 12, 2018, 6:49am UTC](https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361/2 "2018-06-12T06:49:04Z")

</div>

Have a look at the gzip\_lines codec.

---

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 18, 2018, 8:00am UTC](https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361/3 "2018-06-18T08:00:55Z")

</div>

Yeah I tried gzip\_lines codec input plugin but it throws error message,

**A plugin had an unrecoverable error. Will restart this plugin. Plugin: ["/test.txt"], start\_position=\>"beginning", sincedb\_path=\>"/dev/null", codec=\>"UTF-8"\>, stat\_interval=\>1, discover\_interval=\>15, sincedb\_write\_interval=\>15, delimiter=\>"\n"\>**  
**Error: Object: sample.gz is not a legal argument to this wrapper, cause it doesn't respond to "read". {:level=\>:error}**

My LogStash input plugin

file {  
path =\> "/test.txt" (this file contains list of gunzip files)  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
codec =\> "gzip\_lines"  
}

And I also tried path =\> "/sample.gz" directly. but the error was same.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2018, 8:01am UTC](https://discuss.elastic.co/t/how-to-efficiently-parse-log-files-in-logstash/135361/4 "2018-07-16T08:01:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.