Taking array of matching values for one pattern out of Logstash grok

[hym (Hossein Yeganeh Markid)]

Hi all,
I was wondering, is there any way to take an array of matching values out of Grok plugin. let's explain more with an example:

I have a message like "message" = "whatever 0.0.0.0 whatever 1.1.1.1 something 2.2.2.2 ending"
As we can see I have in this example 3 IPs where it can be no IP or n of them. What I am looking for is using something like this %{IPS:ips} and get all IPs in an array in the result.

Any way?

[paz]

In general, this is the default behavior of the grok filter. If you have multiple patterns referring to the same field, it will insert all found values to it as an array.

E.g.

filter {
    grok {
        match => ["message","%{NOTSPACE} %{IPV4:ips} %{NOTSPACE} %{IPV4:ips} %{NOTSPACE} %{IPV4:ips}"]
    }
}

Will give you something like

{
    "@timestamp" => xxx,
            "ips" => [
        [0] "0.0.0.0",
        [1] "1.1.1.1",
        [2] "2.2.2.2"
    ],
      "@version" => "1",
          "host" => "xxx",
       "message" => "whatever 0.0.0.0 whatever 1.1.1.1 something 2.2.2.2 ending\r"
}

But if you're looking for recursive patterns, which I assume you do given you mention you don't know beforehand the amount of repetitions you'll get, there are 2 avenues:

1. If you know the max amount of occurrences, put as many pattern repetitions in your grok pattern and make them optional as grok does not support recursion currently, or
2. Use some custom Ruby code for it, like

filter {
    ruby {
        code => "
            event.set('ips', event.get('message').scan(/[0-9.]+/))
        "
    }
}

Sidenote: Depending on the log structure, other filters might be more appropriate and/or cleaner, like CSV/Dissect/etc.

[hym (Hossein Yeganeh Markid)]

Thanks for the reply, I have tested what you mentioned with Grok debugger but the result is not as expected. Anything I have missed

[paz]

Have you tried running it in Logstash instead?