Filtering specific range of lines from log using Filebeat

Elastic Stack Elasticsearch
1

Hello,

I am working on setting up ELK stack. My use case suggests to filter specific lines from logs using filebeat and send them to Logstash with custom fields appended.
E.g.

Sample Log file:

    Log process started
	Scan started with ID 1 and MID 1
	Processing this event
	Event prod started
	Processing time is 1.2 seconds
	Scan Completed
	Dumping metadata to logs
	Metadata dump finished
	Processing threads stopped

I need to filter the lines from "Scan started with ID 1 and MID 1" to "Scan Completed" (In this log file, 2nd line to 4th last line)
I am expecting the filtered lines should be sent to logstash with ID and MID from logs to be prepended.
E.g.

Scan started with ID 1 and MID 1
ID:1 MID:1 Processing this event
ID:1 MID:1 Event prod started
ID:1 MID:1 Processing time is 1.2 seconds
Scan Completed

I was thinking of using processors like include_fields [when: contains: has_fields: "Scan Started"].
Since, I am new to ELK, I am not very sure how to achieve filtering of specific range of lines.
Can someone please guide me.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.