# Parse Old .gz Log File

**URL:** <https://discuss.elastic.co/t/parse-old-gz-log-file/129628>\
**Category:** Logstash\
**Created:** [April 26, 2018, 9:33am UTC](https://discuss.elastic.co/t/parse-old-gz-log-file/129628 "2018-04-26T09:33:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dee78](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dee78](https://discuss.elastic.co/u/dee78)\
**Post date:** [April 26, 2018, 9:33am UTC](https://discuss.elastic.co/t/parse-old-gz-log-file/129628/1 "2018-04-26T09:33:56Z")

</div>

Hi everybody,  
hope that someone can help me to solve my issue. I've already read all the related topics on this forum but didn't find anything apt to me.  
So I use ELK to parse and process the access.log files of my Squid proxy/cache server and everything works fine for the current access.log file. The problem is that when I uncompress the old log files, rename them as access1.log, access2.log etc.. etc.. in order to pass these old file I didn't get any result.

I setted up also Filebeat to send the output to Logstah on "localhost:5044" and under "Filebeat prospector" the path to be fetched is /var/log/squid/\*.log, then in /usr/share/logstash I have a pipeline-file where logstash takes the input from Filebeat, parse with grok and send the output to Elasticsearch.  
Here ist beat-pipeline config: [https://pastebin.com/mF6UPmGc](https://pastebin.com/mF6UPmGc)

Any idea?  
Thanks in advance to anyone.  
Davide

I'm running Ubuntu Server 16.04 and Elasticasearch, Kibana and Lostash are the 6.2.4.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [May 4, 2018, 11:05am UTC](https://discuss.elastic.co/t/parse-old-gz-log-file/129628/2 "2018-05-04T11:05:47Z")

</div>

We have just released a new version of the file input that supports direct gzip file processing.  
From the Squid rotate docs I assume that you have individual files zipped.

So I assume you want to have a central Logstash instance with beats on the far end squid server.  
`squid(filebeat++) -> LS -> ES`

If you want to give the new code a try you should still use filebeat for tailing the "live" squid logs but you can move or copy the compressed files to the LS machine. Update the file input plugin.`bin/logstash-plugin update logstash-input-file'. Link to [CHANGELOG](https://github.com/logstash-plugins/logstash-input-file/blob/master/CHANGELOG.md#410).

Add the file input to your config and add real paths.

```auto
input {
  beats {
      port => "5044"
  }
  file {
    path => "/path/to/gz/files/*.gz"
    sincedb_path => "/path/to/gz/position-tracking.sincedb"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/path/to/gz/completed.log"
  }
}

```

You may want to add the debugging stdout to you config so you can see what documents are generated.

```auto
  stdout {
    codec => rubydebug
  }

```

If you want to reread the documents again then delete the sincedb file after each run (in my example it is `"/path/to/gz/position-tracking.sincedb"`) or temporarily use `/dev/null`.

Let us know if this new feature works for you.

---

<div class="post-metadata">

**Author:** ![dee78](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dee78](https://discuss.elastic.co/u/dee78)\
**Post date:** [May 4, 2018, 3:32pm UTC](https://discuss.elastic.co/t/parse-old-gz-log-file/129628/3 "2018-05-04T15:32:58Z")

</div>

Hi @guyboertje,  
thank you very much for your reply. I'll test it this week end and I will let you know.  
Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2018, 3:33pm UTC](https://discuss.elastic.co/t/parse-old-gz-log-file/129628/4 "2018-06-01T15:33:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.