Traefik module for filbeat don't parse traefik logs

[e8kor (Evgenii Korniichuk)]

Log message example:
85.115.248.206 - - [19/Jan/2018:06:08:56 +0000] "GET /assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY HTTP/1.1" 200 320 - "Android" 435365 "Host-api-wearerealitygames-com-2" "http://172.25.0.9:4140" 28ms

Grok pattern:
%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( "%{DATA:traefik.access.referrer}")?( "%{DATA:traefik.access.agent}")?(?:%{NUMBER:traefik.access.request_count}|-)?( "%{DATA:traefik.access.frontend_name}")?( "%{DATA:traefik.access.backend_url}")?

error message:
field [agent] not present as part of path [traefik.access.agent]

how can I fix that ?
Thanks in advance

[exekias (Carlos Pérez Aradros)]

Hi @e8kor,

Perhaps you can open a bug report on github? Please specify these details there, also the log output from Filebeat & Elasticsearch would help.

If you want to work on the fix, It looks to me there is some error during ingestion: https://github.com/elastic/beats/blob/master/filebeat/module/traefik/access/ingest/pipeline.json#L37, I think it's failing there, adding "ignore_failure" : true there would probably help.

Best regards

[e8kor (Evgenii Korniichuk)]

thanks, I'll do that asap, also I see that user agent "android" is not parsed. No fields after traefik.access.body_sent.bytes are parsed.

input:
85.115.248.206 - - [19/Jan/2018:06:08:56 +0000] "GET /assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY HTTP/1.1" 200 320 - "Android" 435365 "Host-api-wearerealitygames-com-2" "http://172.25.0.9:4140" 28ms

pattern:
%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} [%{HTTPDATE:traefik.access.time}] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( "%{DATA:traefik.access.referrer}")?( "%{DATA:traefik.access.agent}")?(?:%{NUMBER:traefik.access.request_count}|-)?( "%{DATA:traefik.access.frontend_name}")?( "%{DATA:traefik.access.backend_url}")?

output:
{
"traefik": {
"access": {
"response_code": "200",
"remote_ip": "85.115.248.206",
"method": "GET",
"user_name": "-",
"http_version": "1.1",
"body_sent": {
"bytes": "320"
},
"time": "19/Jan/2018:06:08:56 +0000",
"url": "/assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY"
}
}
}

related GitHub issue: https://github.com/elastic/beats/issues/6111