Custom grok write for my message

Elastic Stack Elasticsearch
1

Hello Everyone I am having following example logs

I want to extract field using filebeat any one can help ?

0.0.0.0 - - [16/Feb/2023:09:54:40 +0000] "POST /api/WebsiteCategory/ProductDesigns HTTP/1.1" 200 95521 "https://www.contrado.co.uk/" "AdsBot-Google (+http://www.google.com/adsbot.html)"%22) "0.0.0.0, 0.0.0.0" "0.104" "0.100" "1102"

2

Hi @dharminfadia,
Is there a specific field you are trying to extract or all of them? Also there might be a module already created for this type of data. What kind of device are the logs coming from?

3

@Wave
Thank you for reply logs coming from nginx and I am using NGINX Module in filebeat for custom logs above log sample is semistructure can you please try to help write grok pattern.

4

Ok try this as a starting point:

%{IP:IP0} - - %{DATA:WHEN} "%{GREEDYDATA:REQUEST}" %{INT:RESPONSE_CODE} %{INT:CODE} "%{GREEDYDATA:REFERRER}"%{GREEDYDATA} "%{IP:IP1}, %{IP:IP2}" "%{NUMBER:NUMBER1}" "%{NUMBER:NUMBER2}" "%{NUMBER:NUMBER3}"

Please modify this for your use case. based upon a sample of 1 data row I don't really know what those fields are. The grok debugger in Kibana (Dev Tools > Grok Debugger) is your friend and what I used to play around with your sample input. Also, see the grok documentation for more info.
Good luck and happy groking.

p.s. You don't say but if you can use more than grok to modify this data I'd personally use dissect first then perhaps grok.

5

@Wave

Thank you for reply this grok is not working properly but now this project on hold thank you for quick response if I got any solution for this I will post here now no any emergancy.

6

Sure thing. I ran it on a 8.6.1 cluster in the Grok Debugger. Like I said just treat it as a place to start. You can just just try with %{IP:IP0} and add pieces back to see what works in your case. I would recommend doing that before throwing anything straight into filebeat. Also, this might be a good use case for an ingest pipeline instead. It can handle grok and provides you with a UI in kibana.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.