Grok positional structure problem

[Ahmad_Moh_d_El_Hasan (Ahmad Moh'd El Hasani)]

Dear sirs

I have a question about the grok plugin:
I have some logs which are built in the following way and my problem is that the order of my fields can mutate from log to log (in the
example here I should extract from all the logs both the "user id" and the "document id"). So I am wondering how I should build the grok
pattern to handle both the logs hopeing there is a better solution of writing two different patterns (or how to write a regex to find
a specific attribute anywhere in the log without knowing before its exact position)

2019-01-15 INFO myclass - mymethod: user id: 12345 custom message 1 document id: 843572309845
2019-01-15 WARN myclass - mymethod1: document id: 43543534 custom message 2 user id: 98589348543 custom message 3 agent id: 98435734

the expected result is:

date: 2019-01-15
level: INFO
class: myclass
method: mymethod
message: user id: 12345 custom message 1 document id: 843572309845
userId: 12345
documentId: 843572309845

date: 2019-01-15
level: WARN
class: myclass
method: mymethod1
message: document id: 43543534 custom message 2 user id: 98589348543 custom message 3 agent id: 98435734
userId: 98589348543
documentId: 43543534
agentId: 98589348543

Looking forward for your answer, thank you.

[Badger]

You could try using

grok {
    break_on_match => false
    match => { "message" => [ "document id: %{NUMBER:documentId}", "user id: %{NUMBER:userId}" ] }
}

[Fabio-sama (Fabio)]

Are only user_id and document_id that can switch order? I mean, is it possible to have something like:

[timestamp] ... agent_id: XXX ... document_id: YYY ... user_id: ZZZ ?

So, for example, with agent_id coming before everything else in the message part of the log?

[Badger]

You could add

"agent id: %{NUMBER:agentId}"

to the array of patterns that grok matches against.

[andres-perez (Andrés Pérez)]

The trick here is that, as break_on_match is set to false, grok won't stop and it will compare the contents of message with every pattern.

Each element of the pattern array will be tested independently against the entire message field, starting from the beginning.