Grok: multiple occurences of pattern

Hello,

I am looking for a solution to do the following: I have a log file where a single log entry can contain multiple key/value pairs that I want to extract. The problem is, I want to extract all occurences in a single entry, not just the first one. (kinda like the "g" option in sed and vim)

For example, a log entry could look like this:

2018-03-28 14:23:56 something something user=foo something something more user=bar

It's trivial to write the grok filter that matches the first occurrence:

grok {
    match => [ "logText", "[^A-Z^a-z^0-9]user=(?<user>[A-Za-z0-9]*)" ]
}

This would give me a document like this:

{
    "user": "foo"
}

What I want though, is this:

{
    "user": ["foo","bar"]
}

Any idea on how to accomplish this?

Perhaps surprisingly, a kv filter will do this.

kv { field_split => " " value_split => "=" }
       "message" => "2018-03-28 14:23:56 something something user=foo something something more user=bar",
          "user" => [
        [0] "foo",
        [1] "bar"
    ]

Interesting, but I'm not looking to extract all possible kv pairs. For instance, a log entry often contains a lot of "uninterestingvariable=something" entries that I do not want to store.

Also, I would like to find something more generic, and not just applicable to key/value pairs. For example, there could be something like IDs in the logfile that have a specific, and easily matched alphanumeric format.

Then I would use a ruby filter to split and then iterate over the fields doing a match on each one.

Interesting, but I'm not looking to extract all possible kv pairs. For instance, a log entry often contains a lot of "uninterestingvariable=something" entries that I do not want to store.

The kv filter's include_keys option can deal with that.

Also, I would like to find something more generic, and not just applicable to key/value pairs. For example, there could be something like IDs in the logfile that have a specific, and easily matched alphanumeric format.

Short of a ruby filter I don't think there's a way of doing that with the standard filters.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.