KV filter on ugly json log

[stanley783]

Hi,

received log in json format, json filter parsed correctly but one field, "details", is in a unfinished/incorrect format lets say and json cant handle it, therefore i applied kv but cant get desired outcome - correct parsing.

below is mentioned field that is parsed from json as this whole value:

id:"KLJASDJLEIFA" creator:"user" expires:16639 description:"Test4" start_request:{artifacts:"Generic.Info" artifacts:"Generic.DiskSpace" specs:{artifact:"Generic.Info" parameters:{}} specs:{artifact:"Generic.DiskSpace" parameters:{}}} condition:{labels:{label:"test"}}

i tought setting recursive to true would solve my issue but no. Outcome field names are like:

bla.specs.{artifact
bla.condition.{labels.{label
bla.start_request.{artifacts

so parts with brackets are making troubles.

in kv filter, among source and target, i have this config:

field_split => " "
recursive => true
whitespace => "strict" 
value_split => ":"

Also, when i had "include_brackets => true" instead of "recursive" it did not work. Did not try both of them at same time tho.
So brackets { } are not considered like they are in json filter. Any ideas how to get desired outcome, if even possible this way? Or would replacing SPACEs with commas, helped and json would be able to do it? My few brain cells are in friday mode already. God bless.

[sudhagar_ramesh]

Hello @stanley783

you can apply the below for the details field

filter
{
mutate{
gsub => ["details", "{", ""]
gsub => ["details", "}", ""]
strip => "details"
}
kv
{
source => "details"
field_split => " "
recursive => true
whitespace => "strict" 
value_split => ":"
}

Keep Posted on how it goes !!! Thanks !!!

[stanley783]

Hi,

thanks for advice but that did not really solved the issue i meant. Removing brackets was not the problem, incorrect parsing logic and field outcome was, when using kv instead of json.

To clarify, check the log from part "start_request" and first two values:
start_request:{artifacts:"Generic.Info" artifacts:"Generic.DiskSpace" specs:{artifact:"Generic.Info" parameters:{}} specs:{artifact:"Generic.DiskSpace" parameters:{}}} condition:{labels:{label:"test"}}

now i get fields without brackets, but other fields that were within/behind the brackets are not considered as part of the previous objects. It parsed as:

details.start_request.artifacts: Generic.Info
details.artifacts: Generic.DiskSpace

(second one should be also in field/object - details.start_request.artifacts, same as first one)

also later on in the log, it parsed details.specs.artifact instead of correct details.start_request.specs.artifact. I wanted to try if i can somehow either edit the log to fit json parsing, or use kv filter options to parse it to some correct degree. Thanks.

[leandrojmp (Leandro Pereira)]

The issue is that your log is neither a valid JSON nor a valid key-value message, you won't be able to solve this using just those filters.

You will probably need to write a custom ruby code and use it to parse it.

Does the format of this details field changes or it has always the same format, with the keys in the same position?

If the keys are always in the same position, then you can solve it using dissect, if the format changes, then I think that the ruby code will be needed.

Also, do you have control on the source of this log and can propose changes?

[stanley783]

Correct, ruby was last resort as i am not that skilled, therefore i wanted to verify that it really cant be easily updated other way.

Details field change based on action type, and based on how many attributes (artifacts) are included within that action (in this case, 2 - Generic.Info , Generic.DiskSpace).

Anyway, i contacted developer/representative of product generating log with this question, and they confirmed it was first intended to be string field, not json (semi-json to help with manual parsing), but they will probably change it to json. In the meantime i probably make some sloppy parsing that will be changed later on.

Thanks for advices.