I know that filebeat has a multiline tool to handle multiline logs. However, the way my log structure is setup I do not think those solutions work exactly the same way. I am using the ELK stack with docker, and for filebeat, I am using the "autodiscover" tool to read docker logs.
When directing my filebeat output to a file, the logs are in the form of
So I think filebeat will look to match a pattern for that entire log, not just the message field. So if I were to try to use multiline, would I have to do something like
multiline.pattern: '^.*\"message\":\"<some_pattern_here>'
multiline.negate: true
multiline.match: after
Also, where would I put this inside the filebeat config. Would I put it one level under the autodiscover, like
filebeat.autodiscover:
multiline.pattern: '^.*\"message\":\"<some_pattern_here>'
multiline.negate: true
multiline.match: after
providers:
....
I also don't just want to move the whole log that doesn't match the matter, I want to move the message field of that log and move it to the end of the message field of the previous log.
Perhaps I'm misunderstanding but it sounds like the multiline messages are encoded within the message field, right? If so, wouldn't there be \n characters in there already? If so, do you even need the multiline.* settings?
It might be helpful if you could copy-paste a small chunk (few lines) from your Docker logs. If it doesn't fit in here, feel free to use https://gist.github.com/ or https://pastebin.com/ and post the link here.
So i'm not totally sure what the multiline plugin is "looking at" to use the pattern. I'm using the autodiscover tool to get docker logs. so if I look at the locations where the docker logs are from, and cat the <docker_id>-json.log file, the lines are in the form
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.