Logstash GROK filter query

Elastic Stack Logstash
1

Hi All,

I'm trying to visualise the statusCode in Kibana and try and get it as a filter. So I originally tried setting filters in logstash with json and mutate however that didn't get me the desired filter. Finally I just wanted to get the data to be visible in Kibana and changed my logstash config as
filter {
grok {
match => { "message" => "%{GREEDYDATA:request}" }
}
json{
source => "request"
target => "logEntry"
}
}

3 sample log file entries:
{"@message":"Client REQUEST","@timestamp":"2017-04-07T14:18:38+0000","@fields":{"data":{"uri":"/status","headers":{"connection":"Keep-Alive","user-agent":"Load Balancer Agent","x-ms-lb-host":"xxx.xx.xx.xx:xxxx","x-ms-lb-monitorstatus":"Up","host":"xxx.xx.xx.xx:xxxx"},"body":{}},"logLevel":"debug","logId":320511,"level":"debug"}}
{"@message":"Client REQUEST","@timestamp":"2016-10-04T09:43:30+0100","@fields":{"uri":"/test/blah/","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","host":"my.server.1","logId":20642,"level":"info"}}
{"@message":"Client RESPONSE:","@timestamp":"2016-10-04T09:43:30+0100","@fields":{"statusCode":200,"logId":20642,"level":"info"}}

I would have expected to see logEntry.@fields.statusCode in Kibana but I do not. However I do see logEntry.@fields.data.uri.

I'm new to this and stuck here and really need some help or pointers to documentation on how to configure the filter so that I can get the filters statusCode, uri logId

My ultimate aim is to be able to plot on a graph all my different statusCode, uri logId on to Kibana dashboard. What am I doing wrong here? What filter setting should I be using?

Any help or tips will be greatly appreciated

2

I would have expected to see logEntry.@fields.statusCode in Kibana but I do not. However I do see logEntry.@fields.data.uri.

Only the last event contains a statusCode field. If you want the uri field and the `statusCode field in the same event you need to join the events. You should be able to use an aggregate filter to join the "Client REQUEST" event with the corresponding "Client RESPONSE" event.

3

Hi Magnus,

Thanks a lot for your reply, I almost gave up on hearing from anyone after 2 days. Thanks again. I just looked into the documentation for Aggregate filter plug-in and it looks like it could do my job.

The field that links the logs is the logEntry.@fields.logId. I am going to look into this in detail now. I'm now also thinking of getting the uri, response time and status code from the example below. If its not too much to ask would you be able to send me the logstash filter configuration I should be using for the sample logs below:
{"@message":"Client REQUEST","@timestamp":"2017-04-10T00:18:45+0000","@fields":{"data":{"uri":"/authorise/users/","headers":{"accept":"%2f","host":"myServer1:9000","connection":"Keep-Alive"},"body":{}},"logLevel":"debug","logId":17186,"level":"debug"}}
{"@message":"Client RESPONSE:","@timestamp":"2017-04-10T00:18:45+0000","@fields":{"data":{"statusCode":200,"body":"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>no"},"logLevel":"debug","logId":17186,"level":"debug"}}

Sorry if I've asked more before even trying this out in detail, anything to save me more time will be really appreaciated. As before thanks again for your response.

4

I've never used the aggregate filter myself so I don't have time to prepare a working example for you.

5

No issues Magnus. Its quite hard trying to get a solution as the logs shown on that page are simple one liners and my log fields are JSON based. Also the issue is that no one else seems to know how to use it for anything more complex than single line text log entries. Thanks again for your time. Appreciate it. Have a good day ahead.

6

Hi @cyberwhiz,

To join request and response into the same event, here's an example configuration :

if [@message] == "Client REQUEST" {
	aggregate {
		task_id => "%{[@fields][logId]}"
		code => "map['uri', event.get('uri')]"
		map_action => "create"
	}
}
else if [@message] == "Client RESPONSE" {
	aggregate {
		task_id => "%{[@fields][logId]}"
		code => "event.set('uri', map['uri'])"
		map_action => "update"
		end_of_task => true
	}
}

The join event will be the 'RESPONSE' one. After aggregate filter, the RESPONSE event will contain the uri field.
If you don't want to keep the REQUEST event, you could add drop {} at end of first if statement.

7

Hi Fabien,

Sorry for the delayed reply, I was away and just got back in this morning. I will give this a try. Thanks a lot

8

OK, tell me if it works fine :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.