Logstash matching on variable fields

[kylegordon]

I have Logstash receiving variable form data over syslog, and I'm having some issues getting Grok to match it. Once I've grabbed the relevant bits of syslog data, I'm left with the following types of messages.

[message]ClusterManager: Check-in complete.[/message] [env]Prod[/env] [exception][/exception] [user][/user] [referrer][/referrer] [user_agent][/user_agent] [method]Quartz.Impl.AdoJobStore.JobStoreSupport+ClusterManager.Run[/method] [line]c:\Program Files x86\Jenkins\workspace\Quartz.NET\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:3677[/line] [thread_id]22[/thread_id] [process_id]1548[/process_id] [process_name]Api.BackgroundService.Host[/process_name] [exception_type][/exception_type]

[message]Processing 1 events for 1146784.[/message] [env]Prod[/env][exception][/exception] [user][/user] [referrer][/referrer] [user_agent][/user_agent] [method]MarketingDatamart.Messaging.Handlers.ActivityBatchHandler.Execute[/method] [line]c:\hudson-it\workspace\Marketing Datamart 02 - Package\Messaging\Handlers\ActivityBatchHandler.cs:62[/line] [thread_id]7[/thread_id] [process_id]9264[/process_id] [process_name]MarketingDatamart.Messaging.Host[/process_name] [exception_type][/exception_type]

Previously we just iterated through a bunch of (?m)[process_name]%{GREEDYDATA:process_name}[/process_name] style matches, but that resulted in _grokparsefailure tags everywhere as understandably some fields were failing. I'd like to merge it all into one line.

My main difficulty is that the fields are defined by the application developer, and may change. My regex-fu is also quite poor! Is anyone aware of a way of iterating through that collection of fields in Logstash and coming out with a list of key/value pairs? In addition, sometimes there are no spaces between field names...

Any advice/thoughts would be great.

[Badger]

Well it looks a little like XML, so I would adjust it to actually be XML.

    mutate { gsub => [ 
        "message", "\[", "<", 
        "message", "\]", ">",
        "message", "^", "<a>", 
        "message", "$", "</a>"
         ] 
    }
    xml { source => "message" store_xml => true target => theXML }

[kylegordon]

That's a great one, @Badger, thank you. I just had to tidy up some embedded characters with more gsubs and it worked

No more massive and easily outdated grok matches!