I am working on setting up ELK stack. My use case suggests to filter specific lines from logs using filebeat and send them to Logstash with custom fields appended.
Sample Log file:
Log process started Scan started with ID 1 and MID 1 Processing this event Event prod started Processing time is 1.2 seconds Scan Completed Dumping metadata to logs Metadata dump finished Processing threads stopped
I need to filter the lines from "Scan started with ID 1 and MID 1" to "Scan Completed" (In this log file, 2nd line to 4th last line)
I am expecting the filtered lines should be sent to logstash with ID and MID from logs to be prepended.
Scan started with ID 1 and MID 1 ID:1 MID:1 Processing this event ID:1 MID:1 Event prod started ID:1 MID:1 Processing time is 1.2 seconds Scan Completed
I was thinking of using processors like include_fields [when: contains: has_fields: "Scan Started"].
Since, I am new to ELK, I am not very sure how to achieve filtering of specific range of lines.
Can someone please guide me.