Grokking works in debugger but differs in Logstash

Elastic Stack Logstash
1

Hello,

I'm using a regex which would remove the first word and parse the rest.

grok {
     match => { "resource" => "/[^/]+(/|)(?<repo>[^/]+)?(/%{GREEDYDATA:resource_path})?" }
      }

Test Messages:

  • /list/Lighter-test-group/xyz/123
  • /list/
  • /list

I've been using https://grokdebug.herokuapp.com/ and https://regex101.com/ to test my regex and grok filter.

For messages,

  • /list/Lighter-test-group/xyz/123 gives us repo value as "Lighter-test-group" which is valid
  • /list/ gives us repo value as null which is valid

but /list gives repo value as "list" which is an invalid value. The correct value needs to be empty or null.

The sites I used for debugging gives me null values for /list but when I run Logstash locally, I see the value as "list". Does anyone know why there's a difference running locally vs using debuggers?

2

They are different code bases supported by different organizations. I would not expect them to stay in sync.

If you want to test grok filters then I would recommend that you do it using grok. Use two windows. In one run logstash with -r on the command line, so that it restarts the pipeline every time the configuration is modified. In the other edit the configuration. I would start with something like either

input { generator { count => 1 lines => [ '/list/Lighter-test-group/xyz/123', "/list" ] } }
filter {
    grok { match => { "message" => "..." } }
}
output { stdout { codec => rubydebug { metadata => false } } }

or

input { file { path => "/home/foo.txt" sincedb_path => "/dev/null" start_position => beginning }
filter {
    grok { match => { "message" => "..." } }
}
output { stdout { codec => rubydebug { metadata => false } } }
3

Thanks for the info.

I've been trying to get the right regex for the above scenario. Any help would be appreciated!