We're currently using Filebeat to ship a log file into Logstash where fields are transformed for searching on in Elasticsearch however I've come into an issue I'm hoping someone here can help with.
So the idea is to store the contents of the log entry in 1 message field in Elasticsearch, here is a sample log file:
01-02-2018 11:00:01 GMT - NOTICE - Cron started. 01-02-2018 11:00:01 GMT - INFO - Processing files from the HTTP server. 01-02-2018 11:00:01 GMT - INFO - About to download and process these files: Array (  => https://www.dummyserver.com/files/file.zip ) 01-02-2018 11:00:01 GMT - DEBUG - Current memory usage pre-loop: 10 mb. 01-02-2018 11:00:06 GMT - NOTICE - Total time taken: 00:05 (M:S) 01-02-2018 11:00:06 GMT - NOTICE - Cron finished with return status 0.
We're using the below filebeat config to store the above log into one message field:
- type: log
multiline.pattern: "(?s)Cron started.*Cron end."
The issue is, we now want to extract log lines that include INFO and DEBUG so only those lines with NOTICE are sent onto logstash but obviously as this multiline is putting the whole log example above into one line it appears as though filebeat can't read the lines with INFO in for example.
I was wondering if there was any option or method for achieving our desired results at the moment?