How to extract and transform unstructured data into fields

My message index field has field1=A big brown wolf, field2=Yet another furry animal, field3=No guarantee how many fields we will see

This is what is indexed:

I'm missing the rest of the words in between the = and the ',' This ends up in the message field
What would be the appropriate kv filter to use so I can have all the words up to the comma','
like so:

field1=A big brown wolf
field2=Yet another furry animal



I do believe this is a limitation of the KV filter (if someone knows a workaround please mention it).
That said, you can practically achieve the same result with some custom Ruby code.

filter {
    ruby {
        code => "event.get('message').split(',').each { |v| field, value = v.strip.split('=') ; event.set(field, value) }"
  1. It first splits the message into chunks separated by commas,
  2. Then it splits each chuck into key-value fields preserving the included spaces (strip just removes leading/trailing ones from the initial split)

Of course for the above to work, the 'message' field should have the exact structure you posted originally, else code adaptations would be needed.

Can't you just set field_split => ","?

Actually it's not so simple .. I have fields with monetary values like this
Amt=100,087,001.098214 so I get Amt=100 because it's taking me up to the comma. How do I further cater for that?

It would help if you posted an actual log line (preferably containing all corner cases).

ClientMapping [counterPartyWID=NOT FOUND, counterParty=NOT FOUND, customerWID=NOT FOUND, customer=NOT FOUND, isCLS=false, isCustomer=true, riskCountry=NOT FOUND], collateralised=false |

So with => ',' I'm getting


etc ... I am missing the second word which is the 'FOUND'

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.