Kv filter with spaces in values

I've got a log that seems a perfect candidate for the kv filter:

msg=Start MID 1242272 ICID 1632662 categorySignificance=/Informational catdt=Web Filtering categoryObject=/Host/Application/Service deviceSeverity=Info rt=1582506453000

and so on ...

but, as you can see, there are values that contain space(s).
The first field, for example, is msg and should contain Start MID 1242272 ICID 1632662, but the filter simply stops at the first space, as for the catdt field, giving:

                         "msg" => "Start",
                          "rt" => "1582506453000",
              "deviceSeverity" => "Info",
                  "@timestamp" => 2020-04-14T15:49:28.031Z,
                       "catdt" => "Web",
              "categoryObject" => "/Host/Application/Service",
                    "@version" => "1",
        "categorySignificance" => "/Informational",

The number of fields may vary.

Is it possible to have the filter to consider as value everything from "=" to the first letter of the next key, where a key is anything matching a "notspace*=" ([^ =]+)= pattern ?

Found a viable answer on stackoverflow:

Essentially, put a separator (comma, in the example) before each key:

mutate {
  gsub => ["message", "(\S+=)", ", \1"]

You could do that in a ruby filter by scaning the string for a regexp and then adding the key / value pairs to the event. Something like this.

Thank you @Badger,
in the end, I resorted to use multiple groks instead of a kv.

I have logs with ten/twenty rows referring to a single logical document, each with dozens of fields (not always all, or in the same order). The average row size is 5kB. The whole file has some millions of rows.

I only need to extract a few fields, so multiple groks seem to be much more efficient.

Now I'm facing the problem of script-updating an array field for these documents.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.