How to update filebeat grok filter

[marcandre (Marc)]

Hello,

I would like to know if you have a link or a step by step guide on how to add more format for filebeat? Looking to add more format for filebeat apache module. Latest version.

Provided Grok expressions do not match field value: [172.243.17.195, 192.88.135.13, 172.31.40.168, 172.31.40.168 127.0.0.1 - - [13/Jun/2019:19:50:02 +0000] "GET

Thank you

[shaunak (Shaunak Kashyap)]

The apache module processes Apache server or error logs using Elasticsearch Ingest Node pipelines. Their definitions can be found here:

• For Apache server logs: https://github.com/elastic/beats/blob/master/filebeat/module/apache/access/ingest/default.json
• For Apache error logs: https://github.com/elastic/beats/blob/master/filebeat/module/apache/error/ingest/pipeline.json

Notice the grok processor in both these pipelines.

If you want to add more formats to this processor definition, you will need to define new lines in the patterns array. For this you will want to make a pull request to the elastic/beats repository. Follow the contribution guidelines listed here: https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html.

Shaunak

[marcandre (Marc)]

Thank you for the answer.

[marcandre (Marc)]

Hello,

Turn out the problem was related to wrong clasification. My previous exampe was flag as Apache errors log and not Access. I Made the changes and now it fix. But I still have an issue with another format. I don't understand what missing.

Message

|start there -> - - - [15/Jun/2019:07:05:36 +0000] "GET /Ski-Switzerland-Arosa-Tschuggen-Grand-Pricing-Info.aspx HTTP/1.1" 301 289 "-" "Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)" vhost=domain.devcloud.acquia-sites.com host=www.domain.com hosting_site=domain pid=3995 request_time=809 forwarded_for="46.229.168.152, 162.158.63.137" request_id="v-f97cfcea-8f3b-11e9-9143-d3a71d7b9165" location="https://www.domain.com/Ski-Switzerland-Arosa-Tschuggen-Grand-Pricing-Info.aspx"

Error

Provided Grok expressions do not match field value: [- - - [15/Jun/2019:07:05:36 +0000] "GET /Ski-Switzerland-Arosa-Tschuggen-Grand-Pricing-Info.aspx HTTP/1.1" 301 289 "-" "Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)" vhost=domain.devcloud.acquia-sites.com host=www.domain.com hosting_site=domain pid=3995 request_time=809 forwarded_for="46.229.168.152, 162.158.63.137" request_id="v-f97cfcea-8f3b-11e9-9143-d3a71d7b9165" location="https://www.domain.com/Ski-Switzerland-Arosa-Tschuggen-Grand-Pricing-Info.aspx\" ]

Thank you,