[Grok] Array of patterns - only first pattern matched

[james64 (Jakub Dubovsky)]

Hello,

based on this documentation I am trying to use grok filter with this configuration:

        grok {
          match => {
            "message" => [
              ' volume_path="%{UNIXPATH:csi_volume_path}',
              ' msg="(?<csi_msg>[^"]*)"',
              ' volume_id=%{UUID:csi_volume_id}',
              ' host_id=%{NUMBER:csi_host_id}',
              ' cmd=%{WORD:csi_cmd}',
              ' args="[(?<csi_cmd_args>[^"]*)]"'
            ]
          }
        }

Here is example message:

time="2020-11-21T15:17:17Z" level=info msg="node get volume stats called" host_id=209881301 method=node_get_volume_stats region=fra1 version=latest volume_id=bafcd0ac-1dbe-11eb-bac9-0a58ac14d147 volume_path="/var/lib/kubelet/pods/bf7f9cdd-d0c0-4b7f-bffb-76e38a590cae/volumes/kubernetes.io~csi/pvc-b4255b85-64c3-47cc-a62c-612164b0eef4/mount"

My log messages matches more then one of these patterns (checked on grokdebug.herokuapp.com). However in document in elastic I can only see result from first pattern (as listed in config). I tried to reorder the patterns in array and restart Logstash. I can see other fields that way. But never more then first one.

Do I miss something or is this a bug? Is there any other way I can match multiple patterns on single field?

Version I use is docker.elastic.co/logstash/logstash:7.9.3

Happy to provide further info. Thanks!

[Badger]

Set the break_on_match option on the grok filter to false if you want it to try all of the patterns.

But why not use a kv filter?

[james64 (Jakub Dubovsky)]

Oh that makes sense thanks!

Yes kv filter would make more sense here. I am trying to use grok because I was tasked with (finding my own sample data and) learning grok filter. It's part of mentoring program. I was simply too lazy to find/make up log data better suitable for grok so here I am...