Unable to parse Grok Filter on Proxy Logs

Elastic Stack Logstash
1

I am attempting to parse and structured a raw proxy data using GROK filter in the ELK stack and I can,t get the timestamp and user agent string in the correct format. Do refer to sample log as follows:

"1488852784.440 1 10.11.62.19 TCP_DENIED/403 0 GET http://xxx.xxx.com/xxx - NONE/- - BLOCK_WEBCAT_12-XXX-XXX-NONE-NONE-NONE-NONE <IW_aud,0.0,-,""-"",-,-,-,-,""-"",-,-,-,""-"",-,-,""-"",""-"",-,-,IW_aud,-,""-"",""-"",""Unknown"",""Unknown"",""-"",""-"",0.00,0,-,""-"",""-"",-,""-"",-,-,""-"",""-""> - L ""http://xxx.xxx.xxx"" 10.11.11.2 - 403 TCP_DENIED ""Streaming Audio"" - - - GET ""Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"" http://xxx.xxx.xxx"

I am using the following filter:
%{NUMBER:timestamp}%{SPACE}%{NUMBER:request_msec:float} %{IPORHOST:src_ip} %{WORD}/%{NUMBER:response_status:int} %{NUMBER:response_size} %{WORD:http_method} (%{URIPROTO:http_proto}://)?%{IPORHOST:dst_host}(?::%{POSINT:port})?(?:%{NOTSPACE:uri_param})? %{USERNAME:user} %{WORD}/(%{IPORHOST:dst_ip}|-)%{GREEDYDATA:content_type}

based on http://grokconstructor.appspot.com, I am able to parse out some of the fields except the timestamp (1488852784.440) and User Agent String. I have tried different Drok default filters on the timestamp but it still shows as numbers.

thank you.

2

I have tried different Drok default filters on the timestamp but it still shows as numbers.

Well... it is a number. If you want to turn it into a human-readable timestamp you need to use a date filter. If that doesn't answer your question, please show us what get now (use a stdout { codec => rubydebug } output) and what you'd like to get instead.

3

hello magnus,

I am getting the following in logstash upon executing the conf file:

"timestamp" => "1488852784.674",
"tags" => [
[0] "_dateparsefailure"

my date filter is as such:

match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]

4

Your timestamp field doesn't contain a yyyy-MM-dd date so this shouldn't be surprising. Use UNIX as your date pattern.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.