# AKAMAI =\> Filebeat =\> Logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 8, 2017, 8:56am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801 "2017-12-08T08:56:57Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Haenschen](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@Haenschen](https://discuss.elastic.co/u/Haenschen)\ **Post date:** [December 8, 2017, 8:56am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/1 "2017-12-08T08:56:57Z")
Hi folks, I'm going to include our Akamai Logfiles in our Elastic Stack. I'm syncing the compressed logs via rsync to a local storage. There are only compressed (gzip) files. As I read, there were some issues using Logstash and gzip files. My filebeat is harvesting the Files properly and sending it to Logstash. But I have some trouble viewing the lines in Kibana. First of all: my configuration: ##################################### cat 02-beats-input.conf input { beats { port =\> 5044 client\_inactivity\_timeout =\> 3600 } } ##################################### cat 15-akamai-cdn.conf filter { if [type] == "akamai-cdn" { grok { match =\> { "message" =\> "%{DATE\_EU:date}\t%{TIME:time}\s%{IP:clientip}\s%{WORD:httpmethod}\t%{PATH:requestedpage}\s%{NUMBER:responsecode}\s%{NUMBER:bytessent}\s%{NUMBER:timetaken}\t%{DATA:csreferrer}\t%{DATA:csuseragent}\t%{DATA:cscookie}" } add\_tag =\> ["akamai-cdn"] } } } ##################################### cat 30-elasticsearch-output.conf output { elasticsearch { hosts =\> ["elasticsearch.xxxxxxxxxxxxxt:9200"] manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } ##################################### This is working fine so far, but the logfile of filebeat contains the following: 2017-12-08T09:50:44+01:00 DBG Publish: { "@timestamp": "2017-12-08T08:50:43.026Z", "beat": { "hostname": "logstash", "name": "logstash", "version": "5.6.5" }, "fields": { "env": "staging" }, "input\_type": "log", "message": "{p\ufffd\u000c\ufffd\u0011q\ufffd\u0001\ufffd^p\ufffd\ufffdD.\u001du\u001b\ufffd#"\ufffdH\ufffd\u0007.", "offset": 1350806, "source": "/data/logs/akamai-cdn/pictures1\_xxxxxxxx\_de\_xxxxx5.esw3c\_S.201711110000-2400-22.gz", "tags": [ "akamai-cdn", "akamai-filebeat" ], "type": "log" } and if you look at this in Kibana it shows the following "message" : ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/5/1537e9c5a7fd4e9504efbf6d5e60a153caf41880.png) Does anybody have a solution or an idea to solve this ? best regards Hans ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2017, 9:07am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/2 "2017-12-08T09:07:45Z")
I don't think Filebeat can handle gzipped files either, and it looks like the message is indeed in binary form. You probably need to uncompress the files before Filebeat can correctly pick them up. ---
**Author:** ![Haenschen](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@Haenschen](https://discuss.elastic.co/u/Haenschen)\ **Post date:** [December 8, 2017, 9:41am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/3 "2017-12-08T09:41:13Z")
I've tried the following: cat 01-akamai-files.conf input { file { max\_open\_files =\> 65000 id =\> "akamai-cdn" type =\> "akamai-cdn" path =\> "/data/logs/akamai-cdn/\*" start\_position =\> "beginning" sincedb\_path =\> "gzip" codec =\> "gzip\_lines" } } - same result... ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3b8a4ed5e37c9c4fad95e39f605ed0042abc9991.png) even if you decompress the files and exclude every \*.gz files ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2017, 5:59pm UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/4 "2017-12-11T17:59:14Z")
What is the format of the log files? Do they contain binary data? ---
**Author:** ![Haenschen](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@Haenschen](https://discuss.elastic.co/u/Haenschen)\ **Post date:** [December 12, 2017, 5:45am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/5 "2017-12-12T05:45:03Z")
Hi Christian, there are only gzipped textfiles, no binary data. That's what I'm wondering.... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2017, 9:09am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/6 "2017-12-12T09:09:29Z")
If you unzip one of the files, can you provide a few sample lines? What happens if you manually unzip one of the files and read it with the `codec => "gzip_lines"` removed? ---
**Author:** ![Haenschen](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@Haenschen](https://discuss.elastic.co/u/Haenschen)\ **Post date:** [December 12, 2017, 10:42am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/7 "2017-12-12T10:42:08Z")
Hi there, thank you for your help, I finally got it However, logstash didn't exclude the \*.gz files in the destination directory, you have to copy it anywhere and gzip -d it to parse the logfiles I can write a detailed HOWTO if you would like, but I don't think it's necessary. This can be closed now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 9, 2018, 10:42am UTC](https://discuss.elastic.co/t/akamai-filebeat-logstash/110801/8 "2018-01-09T10:42:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.