How to filter my logs

[kimfut (kim)]

hello everyone , i am trying to filter my logs to something more structured , mostly the message with url part , i tried a lot of grok filter that i found online but everytime i got " _grokparsefailure" in my logs and nothing wad filtered.

there is example of my logs :

can you help me please to define which filter can do the job

thank you

[Christian_Dahlqvist]

Have a look at this blog post. It will show you how to work with Logstash and go about creating grok configuration.

[MrNerd]

@kimfut I rode this guide and for me was the best one to understand how to filer

Do you grok Grok? | Elastic Blog

Regards

MrNerd

[inhinyera16 (Archelle Pagapulan)]

Maybe your grok pattern doesnt match at all and so it fails.
You need to atleast make sure that the logs are uniform in pattern.

Also could you provide us your grok pattern?

[kimfut (kim)]

that's my grok pattern :

filter {

   grok {
   match => {"message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} "}


   }

}

[arkady_renko (Mehmet Sezgin)]

Message field in attached image has date string between ip address and method. But your grok filter doesnt have any date field.

Your log in message looks like apache log.
İf it is, you can use built-in COMBINEDAPACHELOG pattern.

[kimfut (kim)]

@arkady_renko

use built-in COMBINEDAPACHELOG pattern.

thanx , but when i use :

 grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
   }

it's the same problem :
tags: beats_input_codec_plain_applied, _grokparsefailure that what displayed on kibana

[kimfut (kim)]

hello everyone , I was finally able to get a filter for my logs , this is an example in Json format :

{
"_index": "beat-test",
"_type": "doc",
"id": "s9jhl2MBN2K6P6dYqwl",
"_version": 1,
"_score": null,
"_source": {
"verb": "GET",
"host": "monetoring.com",
"bytes": 184693,
"message": "11.115.98.127 - - [04/May/2018:16:39:20 +0200] "GET /WebServiceRequestProxyCenter/api/request-url?service=density_analyze%2C+siniat%2C+isolation+phonique&url=https%3A%2F%2Fwww.google.fr%2Fsearch%3Fq%3Disolation%2Bphonique%26ie%3DUTF-8%26oe%3DUTF8%26hl%3Dfr%26start%3D0%26num%3D100&country=fr HTTP/1.1" 200 184693",
"ident": "-",
"clientip": "11.115.98.127",
"@timestamp": "2018-05-04T14:39:20.000Z",
"geoip": {
"ip": "11.115.98.127",
"latitude": 48.8582,
"timezone": "Europe/Paris",
"country_name": "France",
"continent_code": "EU",
"country_code2": "FR",
"country_code3": "FR",
"location": {
"lon": 2.3387000000000002,
"lat": 48.8582
},
"longitude": 2.3387000000000002
},
"httpversion": "1.1",
"beat": {
"version": "6.2.4",
"name": "monetoring.com",
"hostname": "monetoring.com"
},
"@version": "1",
"offset": 15692839,
"auth": "-",
"source": "/home/hakim/monetoring.com.access.2018-05-04.log",
"timestamp": "04/May/2018:16:39:20 +0200",
"response": 200,
"tags": [
"beats_input_codec_plain_applied"
],
"request": "/WebServiceRequestProxyCenter/api/request-url?service=density_analyze%2C+siniat%2C+isolation+phonique&url=https%3A%2F%2Fwww.google.fr%2Fsearch%3Fq%3Disolation%2Bphonique%26ie%3DUTF-8%26oe%3DUTF8%26hl%3Dfr%26start%3D0%26num%3D100&country=fr"
},
"fields": {
"@timestamp": [
"2018-05-04T14:39:20.000Z"
]
},
"sort": [
1525444760000
]
}

but it's not very interesting for me , because what I would like to have , it's this part :

"request": "/WebServiceRequestProxyCenter/api/request-url?service=density_analyze%2C+siniat%2C+isolation+phonique&url=https%3A%2F%2Fwww.google.fr%2Fsearch%3Fq%3Disolation%2Bphonique%26ie%3DUTF-8%26oe%3DUTF8%26hl%3Dfr%26start%3D0%26num%3D100&country=fr"

more split to extract "/WebServiceRequestProxyCenter/api/request-url" apart

also to delete all special caracters like we see here : % &

so if you have suggest , help me please

[sana1]

Please give me grok filter for my log pattern, I have a log directory belong to same log server and each file has different data and format but the values are in key value pair. Please gimme way out to parse these logs

right now my filter is

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}

and my data is

"GET / HTTP/1.1
Host: yahoo.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: B=fh4crrpd9sf94&b=3&s=e1; ucs=lnct=1525329090; HP=1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

HTTP/1.1 301 Moved Permanently
Date: Thu, 07 May 2017 09:48:52 GMT
Connection: keep-alive
Via: http/1.1 media-router-fp1012.prod.media.gq1.yahoo.com (ApacheTrafficServer [c s f ])
Server: ATS
Cache-Control: no-store, no-cache
Content-Type: text/html
Content-Language: en
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=2592000
Location: https://www.yahoo.com/
Content-Length: 8"