My task is to introduce log analysis to an environment where I am a drop-in replacement for Syslog and I do not have the option of asking clients to use a different shipper.
In this environment logs are forwarded to my machine via Syslog from multiple different systems running a variety of applications. I listen to the Syslog stream with Logstash, and send to ElasticSearch.
I need to match each line to any one of a number of log format patterns, and once matched, parse out the appropriate fields and set meta identifying type.
I also need to recognize and deal with different types of multiline messages, appending only the syslog-message portion to the previous syslog-message and not the entire line including syslog preamble. This appears to be very tricky if possible at all, possibly requiring a combination of multiline, match and mutate. My search for information on this has been complicated by the evolution of the multiline filter to a codec, making it hard to tell whether some advice is currently relevant.
As this use case seems to me to be a nut that a few people have likely cracked, I'm looking for pointers and links that will set me on the right path.