Grokking works in debugger but differs in Logstash


I'm using a regex which would remove the first word and parse the rest.

grok {
     match => { "resource" => "/[^/]+(/|)(?<repo>[^/]+)?(/%{GREEDYDATA:resource_path})?" }

Test Messages:

  • /list/Lighter-test-group/xyz/123
  • /list/
  • /list

I've been using and to test my regex and grok filter.

For messages,

  • /list/Lighter-test-group/xyz/123 gives us repo value as "Lighter-test-group" which is valid
  • /list/ gives us repo value as null which is valid

but /list gives repo value as "list" which is an invalid value. The correct value needs to be empty or null.

The sites I used for debugging gives me null values for /list but when I run Logstash locally, I see the value as "list". Does anyone know why there's a difference running locally vs using debuggers?

They are different code bases supported by different organizations. I would not expect them to stay in sync.

If you want to test grok filters then I would recommend that you do it using grok. Use two windows. In one run logstash with -r on the command line, so that it restarts the pipeline every time the configuration is modified. In the other edit the configuration. I would start with something like either

input { generator { count => 1 lines => [ '/list/Lighter-test-group/xyz/123', "/list" ] } }
filter {
    grok { match => { "message" => "..." } }
output { stdout { codec => rubydebug { metadata => false } } }


input { file { path => "/home/foo.txt" sincedb_path => "/dev/null" start_position => beginning }
filter {
    grok { match => { "message" => "..." } }
output { stdout { codec => rubydebug { metadata => false } } }

Thanks for the info.

I've been trying to get the right regex for the above scenario. Any help would be appreciated!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.