HSL F5 Sample Grok

Elastic Stack Logstash
#1

Hi there, can anyone help me with this logstash Grok from the sample configuration on the elastic site?

so im using the F5 template from the example here https://www.elastic.co/blog/f5-high-speed-logging-with-elastic-stack and also the example grok filter as is seems to have pretty much everything i need, however it seems to fail as below and i get a _grokparsefail

So sample data from my F5:

12.123.123.254 [13/Feb/2016:16:04:13 +0000] 123.16.1.23 /SITE/SITE_SSLV2 /SITE/SITE_SSLV2_pool 123.16.1.156 443 '/' 'GET / HTTP/1.1' 200 21003 217 'https://w2.website.co.uk/Planner/Planner' 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36'

Example filter which should match:

%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}

However it only seems to get this far before error

%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port}

So this bit \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}

Doesn’t seem to fit this bit'/' 'GET / HTTP/1.1' 200 21003 217 'https://w2.website.co.uk/Planner/Planner' 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36'

even though this is direct from the example.

I dont quite grok these grok filters, could anyone point me in the right direction?

#2

So i fixed the grok using http://grokconstructor.appspot.com/do/match#result

so it now reads %{IP:clientip} [%{HTTPDATE:timestamp}] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} "%{DATA:path}" "(%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion})" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}

however when i add this to my config

filter {
  if [type] == "f5-access" {
    grok {
        match => { "message" => %{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}" \"(%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion})" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent} }
    }
  }
  if [type] == "f5-access" {
    geoip {
        source => "clientip"
    }
  }
}

i get an error

# sudo service logstash configtest
Error: Expected one of #, ", ', -, [, { at line 30, column 33 (byte 339) after filter {
  if [type] == "f5-access" {
    grok {
        match => { "message" =>
#3

Why have two if's that are the same, just merge them.

#4 
    match => { "message" => %{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}" \"(%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion})" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent} }

Comments:

  • The sample log line contains single quotes but your grok expression contains double quotes.
  • The grok expression needs to be a quoted string: match => { "message" => "grok expression here" }.
  • You really should avoid having more than one DATA pattern in the same expression. Depending on the rest of the expression and the input data you could get very weird matches. In this case I suggest you use NOTSPACE instead.
#5

Hi, i still seem to be having real problems with this, is there somewhere logstash caches its filters and im not seeing an update after a service restart?

i changed my pattern in the F5 to remove " from the output and replaced with ' and changed the DATA tags to NOTSPACE tags so my output is now:

123.123.123.20 [16/Feb/2016:12:52:42 +0000] 123.123.1.23 /aaa/aaa_SSLV2 /aaa/aaa_SSLV2_pool 123.123.1.158 443 '/api/planneremployee' 'GET /api/planneremployee?AuthorisationToken=2f368652-c1c2-4828-bc7a-f00a0479258f&end=2016-07-01T00:00:00.000Z&start=2016-06-01T00:00:00.000Z HTTP/1.1' 200 4686 602 'https://www.aaaa.co.uk/Planner/Planner#?date=01-06-2016&mode=timeline' 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)'

and my filter is now:

%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{NOTSPACE:virtual_name} %{NOTSPACE:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \'%{NOTSPACE:path}' \'%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}' %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}

the grok seems to work perfectly when I run in constructor, it now starts fine in logstash but i'm still getting _grokparseerrors on my output rather than the desired fields.

this is the full filter conf

filter {
    if [type] == "BIGIP" {
            grok {
                    match => { "message" => "%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{NOTSPACE:virtual_name} %{NOTSPACE:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \'%{NOTSPACE:path}' \'%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}' %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}" }
                    add_tag => ["grok_match", "F5"]
                    remove_tag => [ "_grokparsefailure" ]
                    remove_field => [ "message" ]
                    }
            }
    }
#6

Hi, i still seem to be having real problems with this, is there somewhere logstash caches its filters and im not seeing an update after a service restart?

No, but make sure you don't have any unexpected files in /etc/logstash/conf.d. If you start Logstash by pointing it to that directory (or some other directory) Logstash will read all files in there.

the grok seems to work perfectly when I run in constructor, it now starts fine in logstash but i'm still getting _grokparseerrors on my output rather than the desired fields.

Be systematic. Start with the absolutely simplest expression, %{IP:clientip}. If that works, tack on another token. Continue until is stops working.

#7

cool, thanks for the quick response

#8