How to extract and transform unstructured data into fields

[sentinel]

My message index field has field1=A big brown wolf, field2=Yet another furry animal, field3=No guarantee how many fields we will see

This is what is indexed:
field1=A
field2=Yet
field3=No

I'm missing the rest of the words in between the = and the ',' This ends up in the message field
What would be the appropriate kv filter to use so I can have all the words up to the comma','
like so:

field1=A big brown wolf
field2=Yet another furry animal

etc.

thanks

[paz]

I do believe this is a limitation of the KV filter (if someone knows a workaround please mention it).
That said, you can practically achieve the same result with some custom Ruby code.

filter {
    ruby {
        code => "event.get('message').split(',').each { |v| field, value = v.strip.split('=') ; event.set(field, value) }"
    }
}

1. It first splits the message into chunks separated by commas,
2. Then it splits each chuck into key-value fields preserving the included spaces (strip just removes leading/trailing ones from the initial split)

Of course for the above to work, the 'message' field should have the exact structure you posted originally, else code adaptations would be needed.

[magnusbaeck (Magnus Bäck)]

Can't you just set field_split => ","?

[sentinel]

Actually it's not so simple .. I have fields with monetary values like this
Amt=100,087,001.098214 so I get Amt=100 because it's taking me up to the comma. How do I further cater for that?

[paz]

It would help if you posted an actual log line (preferably containing all corner cases).

[sentinel]

ClientMapping [counterPartyWID=NOT FOUND, counterParty=NOT FOUND, customerWID=NOT FOUND, customer=NOT FOUND, isCLS=false, isCustomer=true, riskCountry=NOT FOUND], collateralised=false |

So with => ',' I'm getting

counterPartyWID=NOT
customerWID=NOT
customer=NOT

etc ... I am missing the second word which is the 'FOUND'