I have a use case , logs are coming into logstash but due to some failure their is a time elapse between two timestamps for example each log has time period between it of 2 minutes and unexpectedly their is 10 minutes gap and again log are coming now i want to analyse that gap , maybe due to some JVM failure that time gap came so that starting timestamp will have that message of failure so i can analyse it . So is there a way in logstash that i can see that gap and analyse log having timestamp just before the gap came. Is there any filter plugin for it or we can perform arithmetic operation in logstash??
Perhaps the elapsed filter can help?
can elapse tell time difference between each event in log? because i think it just tell time elapse between whole log running . can you give some example of it because i am not able to understand whats given in docs. should i specify task_id there or something else. Can you tell me that please?
The elapsed filter tracks the time between start/end events matching a particular pattern. Perhaps that pattern can be configured to match all messages and you get the time between each message? Otherwise I don't know.
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601} START id: (?<task_id>.*)"]
add_tag => [ "taskStarted" ]
}
}
here at match is it right? can we write like this because i am getting _grokparsefailure
task_id is field from my logs or i make new ?
I don't know what your messages look like so it's impossible to judge if the grok expression is correct.
I just found out elapsed wont help me as it wont give difference of every timestamp in log file, is their any other way , script filter of elasticsearch or anything ?? i just need a way so that i can plot difference of timestamps in a bar chart , so that maximum difference will automatically show that something is weird in logs .