Hi,
I am using logstash to read a particular type of logs from multiple input files like
file {
path => "C:/Users/Documents///*.txt"
start_position => "beginning"
}
Each file contains many lines in which each line is taken and the filters are applied and pushed into the kafka endpoint.
The intent is to setup a logging mechanism to monitor the logstash. So that it will be easy to debug when there is problem in data getting pushed into kafka endpoint.
The expected output in the log file for each file processed should be :
File Path ------ No of records processed in that file
FIle Path ------ No of records processed in that file
The logstash is configured as a service, so that the logs should be continuously generated for the all files being processed by the logstash.
I tried print the path by storing it in a variable using GROK filter. But the path gets printed as many times the number of records in the file. (i.e) 1000 times if there are 1000 lines in the file. This is for a single file but I have multiple input files so the logs are very very huge.
And also i tried to use metrics plugin, but it just gives out the count irrespective of the path of the file.
But I want an output like combination of both the results.
Question :
How can I generate the logs in the format expected?
Kindly help me in solving the issue.
Thanks,
Aravind.