Logs Timestamp

[PritiS]

Hi ,

I am trying to fetch total timings taken by a Job from Status A to Status B from logs, extracted Log timestamp column,status, jobid from logs in Kibana, How can we extract the total time taken by the Job ID and visualize this in Kibana graph,
Can you please provide support..

Thank you,
Priti

[LeeDr (Lee Drengenberg)]

Hi Priti,

This is actually rather difficult to do in Kibana. One way is by getting the data into an entity-centric index in Elasticsearch. What this means is that instead of having a time-based index where each event is indexed into Elasticsearch you would have one document per JobID and those documents would get updated with the time for each status and duration.

Calculate time difference between two different record timestamps

So I am logging a bunch of logs of a big and mediumly complex workflow. This workflow is a graph and can take different paths based on the inputs or a particular workflow step's output. When the workflow is created a workflow id is generated, and is logged with the timestamp. Now I am taksed with calculating the average time it takes for the workflow to reach a workflow step X. What is the best way to achieve this? In my research so far, I have come across elapsed filter and this suggestion o…

Here's the only "solution" I know of that could work directly in Kibana using Advanced JSON field in a Kibana visualization. But if you have a large index, the performance could be a problem;

Display concurrency in data on Kibana

You can actually do this if you query Elasticsearch directly by using a script on a date_histogram aggregation which adds the call document to all the buckets from the start_time to start_time + duration. The following gist contains a Sense recreation that details what I mean: https://gist.github.com/jpountz/cebb8452648c36099cee As for doing this in Kibana, I know it's possible to run scripts in Kibana for some actions but I'm not sure if this would be possible, maybe someone more familiar with…

Related links;

Calculate the time difference between consecutive documents

HI @MariumHassan this is my script input { elasticsearch { hosts => "http://192.168.55.213:9200" index => "test_index" } } filter { aggregate { task_id => "%{logTimestamp}" code => " map['logTimestamp'] = event.get('logTimestamp'); map['timeDifference']= (Time.parse(event.get('logTimestamp')).to_f - Time.parse(map['previousTime']).to_f).round(4); map['previousTime'] = event.get('logTimestamp'); " } } output { elasticsearch { document_id => "%{logTimestamp}" document_typ…

How to calculate time difference between 2 log lines for a unique ID

I need to calculate the difference in time between 2 log lines for http code 200. The difference should be done for a unique id. Please find the log line below. 2018/01/16 00:13:44.890 [HCServiceImpl] [**qtp5720769-243**]: AUDIT- INFO: Request received : /mdp/content?sourceType=cid&pset=mdp%3Ano-presentation&filter%3AcontentId=cid%3A%2F%2Fprogramid%253A%252F%252F2090543776%23programid%253A%252F%252F1264605229 2018/01/16 00:13:44.897 [HCServiceImpl] [**qtp5720769-243**]: AUDIT- INFO:…

Calculate time difference between two logs with uniqueid

hi there is there anyway to calculate time difference between two logs with uniqueid? eg i create fields log_timestamp, log_type, user_id, mobile_no, etc., below logs 2018-09-19 09:24:57,113 (null) 21 WARN : Test :: vh1o4ctmdpn3aft53o5nx0nh :: User Req : <UserRequest><version>9.7</version><userID>458458000000018</userID><userCode>2C10000299</userCode><desc>123456</desc><desc>123456</desc><mobileNo></mobileNo></UserRequest> 2018-09-19 09:24:57,453 (null) 21 WARN : Test :: vh1o4ctmdpn3aft53o5n…

[shwesinhan (ssh)]

Hi PritiS

Do you use logstash and realtime log shipping? If so, lucky you.

I solved by using logstash grok filter and elapsed filter.
For instance, let me share how to do

• make start point for Status A log
• make end point for Status B log
• then connect start point and end point using by unique id (eg: jobid)

if "StatusA" in [message] {
grok {
match => { "message" => [ "grok pattern for your StatusALog. Let's say you will get status, jobid"]}
add_field => { "log_type" => "StatusA" }
add_tag => ["status_a"]
}
}

else if "StatusB" in [message] {
grok {
match => { "message" => [ "grok pattern for your StatusBLog. Let's say you will get status, jobid"]}
add_field => { "log_type" => "StatusB" }
add_tag => ["status_b"]
}
}

elapsed {
start_tag => "status_a"
end_tag => "status_b"
unique_id_field => "jobid"
periodic_flush => true
timeout => 2100
}

You will see elapsed_time, elapsed_timestamp_start fields at StatusB log in kibana. elapsed_time is different timestamp of StatusA and StatusB. elapsed_time field is float type. It's cool.
Then visualize it. I use TimeLion visualization. Here is my sample.
Y-axis : elapsed_time (in sec)
X-axis : time

But plz note that elapsed_time values are only accurate at same CPU.
If pipeline.workers: 1, the result will accurate.
If pipeline.workers: 2, the result won't accurate.