So I have a log file that is taken as an input by filebeat which is then outputted to logstash and from there to Elasticsearch. I need to calculate the time required/elapsed from the time I start filbeat and its reading process till the time it reaches Elasticsearch.
The pipeline is logfile-->filebeat-->logstash-->els. I need to find time elapsed from filbeat to els. I'm new to ELK and don't know how to use grok or similar features yet.
I have configured filebeat.yml as follows:
filebeat.inputs:
- type: log paths:
- /home/user/us/logs/lgfile.json output.logstash: hosts: ["localhost:port"]
and logstash conf file as:
input {
beats {
port => "5044"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:port"]
index => "elkpipego"
}
}
So far I have manually done this time calculation using a stopwatch from the time logstash reads the first msg and then used the difference with the timestamp of the last message. But this is clearly not so accurate and I would like to know if there's a way using any feature/grok/tool like jmeter/esrally or anything where I could accurately find the exact time elapsed till the last message is stashed on Elasticsearch? TIA