Running grok against an HTTP source

Elastic Stack Logstash
1

Hi,

I'm trying to recieve HTTP GET requests via the HTTP input and then run grok and a ruby script on the request_uri in order to extract the paramters and their values.
My agent configuration is as follows:

input
{
  http {
  }
}

filter
{
  grok {
    patterns_dir => "/opt/logstash/patterns"
    match => {"request_uri" => "{URIPATH:path}\?%{PARAMS:params}"}
  }
  ruby {
    code => "
        fieldArray = event['params'].split('&')
        for field in fieldArray
            if (field != '')
              result = field.split('=')
              event[result[0]] = result[1]
            end
        end
    "
  }
}

output
{
  stdout { codec => rubydebug }
}

custom pattern is:
PARAMS [A-Za-z0-9$.+!'|(){},~@#%&/=:;_?-[]<>]

If I try to run curl -XGET 'http://localhost:8080/l/?a=1&bc=2&cdf=3' for the sake of the example, I get grokparsefailure.
Since the grok pattern should be fine (grokdebug shows no issues with it), I tend to believe it's something wrong with how I'm trying to get request_uri. In the response I get -

{
       "message" => "",
      "@version" => "1",
    "@timestamp" => "2016-09-15T13:22:32.695Z",
          "host" => "0:0:0:0:0:0:0:1",
       "headers" => {
         "request_method" => "GET",
           "request_path" => "/l/",
            "request_uri" => "/l/?a=1&bc=2&cdf=3",
           "http_version" => "HTTP/1.1",
        "http_user_agent" => "curl/7.29.0",
              "http_host" => "localhost:8080",
            "http_accept" => "*/*"
    },
          "tags" => [
        [0] "_grokparsefailure",
        [1] "_rubyexception"
    ]
}

I can see that request_uri is nested inside headers, which is what I believe causes my problem.
Am I on the right track? Completly off?

2

I can see that request_uri is nested inside headers, which is what I believe causes my problem.

Yes. You need to reference the field as [headers][request_uri] in your grok filter. See Accessing event data and fields | Logstash Reference [8.11] | Elastic.

1 Like