Grok filter pattern not working

[BeMoore]

I must be going mad today, i can't get this filter to work.
I extracted a single log line from an azure activity log, so i parse it through grok debugger both in kibana and herokuapp and i can't get it to work. so i split it out and chopped it down to try and isolate why its not working

'createdDateTime': '2019-12-11T09:06:47.5489252Z', 'userDisplayName': 'Some User, 'userPrincipalName': 'some.user@custom.domain

and the pattern

(\')%{TIMESTAMP_ISO8601:timestamp}(\'),\s(\')%{USER:userDisplayName}(\'),\s(\')%{WORD:principlename}(\')

and in both debuggers, it doesn't return anything, yet the individual usages work....
i know its friday, and its been a long slog this week, but im not surely missing something so simple here am i ?

[ITIC (Jordi)]

Hi

It might be a typo, but your line is missing some "'" that you are trying to parse in your pattern.

Otherwise, I think your match, assuming your line is in message, could be something like this:

"messge" => "'createdDateTime': '%{TIMESTAMP_ISO8601:timestamp}', 'userDisplayName': '%{USER:userDisplayName}, 'userPrincipalName': '%{GREEDYDATA:principlename}"

Hope this helps

[BeMoore]

@ITIC

"messge" => "'createdDateTime': '%{TIMESTAMP_ISO8601:timestamp}', 'userDisplayName': '%{USER:userDisplayName}, 'userPrincipalName': '%{GREEDYDATA:principlename}"

Nope, thats not working either. However its got something to do with , followed by a space. The parsing breaks there.
with the data i should be able to parse it with just the following line

'createdDateTime': '%{TIMESTAMP_ISO8601:timestamp}', 'userDisplayName': '%{USER:userDisplayName}, 'userPrincipalName': '%{GREEDYDATA:principlename}

but it doesn't

Yet each one works on its own with its own data, except when inline with , and a space. in my logic, including the, and a \s should eliminate this but its not, hence banging my head on the table repeatedly this morning each time this doesn't work.!

this should work

but it dosn't

this works without , and a space

[ITIC (Jordi)]

Hi

Looking at your pictures I see the "'" are not missing in your line, so the filter should be

"messge" => "'createdDateTime': '%{TIMESTAMP_ISO8601:timestamp}', 'userDisplayName': '%{USER:userDisplayName}', 'userPrincipalName': '%{GREEDYDATA:principlename}'"

This should work. If it doesn't I'll start also banging my head on the table

[BeMoore]

Its a space issue with the grok parsing.
two images.... one stock as per your recommendation ( which we'd already tried )
second with alteration to the username and inserting . between some and user

So im on 7.5 ES fully updated at the beginning on this week and in my mind, this means something has gone wrong with the existing patterns that %{USERNAME} makes use of, or something else is foobar'd in respect to how it parses "spaces" in data.
This guy has exactly same issue

_grokparsefailure as soon as space in field

The first pattern does not work because inside square brackets period does not mean "any character", it means a literal period. The other two work for me. For example, input { generator { count => 1 lines => [ '"foo bar"' ] } } filter { grok { match => { "message" => "\"(?<ip2>[\S\s]+)\"" } } } output { stdout { codec => rubydebug { metadata => false } } } produces "ip2" => "foo bar", "message" => "\"foo bar\"",

[BeMoore]

[BeMoore]

and doesn't appear to be anything related in the 7.5.0 change log for this.

[BeMoore]

btw... to get around this, %{DATA:username} works

[ITIC (Jordi)]

You are right! Neither USER nor USERNAME allow for spaces in their definitions (see https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns), so you have to use DATA or GREEDYDATA or similar.

I didn't catch that one!