Hi everyone,
I have some problem creating a logstash configuration file to parse akamai logs. In my case, Akamai logs are in a FTP server and they don't have the typical .log extension, because they're all zipped (.gz).
So I've started to create this configuration file:
input {
_ file {_
_ path => "/home/akamailogs/*.gz"_
_ start_position => "beginning"_
_ codec => "gzip_lines"_
_ type => "akamai"_
_ }_
}
filter {
_ grok {_
_ match => { "message" => "%{IP:clientip}\t%{HTTPDATE:timestamp}\t%{WORD:verb}\t%{URIPATHPARAM:request}\t%{NUMBER:httpversion}\t%{NUMBER:response:int}\t%{NUMBER:bytes:int}\t%{URI:referrer}\t%{QS:agent}\t%{QS:cookie}\t%{URIPATHPARAM:cdn\t%{NUMBER:cdnumb:int}" }_
_ }_
_ date {_
_ match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]_
_ }_
}
output {
_ elasticsearch {_
_ hosts => ["http://****"]_
_ user => "elastic"_
_ password => "changeme"_
_ index => "akamai_logs-%{+YYYY-MM-dd}"_
_ }_
_ stdout { codec => rubydebug }_
}
The following example is the log file that I need to parse:
91.24.8.111 - - [25/May/2017:00:00:00 +0000] "GET /streaming/g244569.mp4?vbegin=3174&token=exp=14956112167%7eacl=/streaming/g21343339.mp4%3Fvbegin=3174*%7ehmac=454fc7e6b547e40656e547efsdasda85ad3b HTTP/1.1" 000 5470467 "-" "AppleCoreMedia/1.0.14 (iPhone; U; CPU OS 10_3_1 like Mac OS X; it_it)" "-" cdn1.test.com 1707 "-"
and this is the grok filter I can't run...
"%{IP:clientip}\t%{HTTPDATE:timestamp}\t%{WORD:verb}\t%{URIPATHPARAM:request}\t%{NUMBER:httpversion}\t%{NUMBER:response:int}\t%{NUMBER:bytes:int}\t%{URI:referrer}\t%{QS:agent}\t%{QS:cookie}\t%{URIPATHPARAM:cdn\t%{NUMBER:cdnumb:int}"
Any advice? 