Measuring the taking time

[shwesinhan (ssh)]

hi everyone!

is there any way to do measuring the time taken by logstash pipeline processing (input cloudwatch logs > filter some patterns > output to aws elasticsearch)????
i am using logstash v6.1.1 and not include x-pack in kibana.

plz, point out to me
thank you

[guyboertje]

There is an API you can use CURL or similar with. https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats

[shwesinhan (ssh)]

thanks @guyboertje

could you explain about "duration_in_millis" and "queue_push_duration_in_millis"

which one is for input/filter/output duration time?

2. i am not clear duration_in_millis is added with previous value or new value?
   eg:

{
"host" : "UAT-LOGSTASH-01",
"version" : "6.1.1",
"http_address" : "127.0.0.1:9600",
"id" : "4f8524dc-38ac-4f3e-9563-bf965573761d",
"name" : "UAT-LOGSTASH-01",
"events" : {
"in" : 2000,
"filtered" : 2000,
"out" : 2000,
"duration_in_millis" : 4284,
"queue_push_duration_in_millis" : 620
}
}

{
"host" : "UAT-LOGSTASH-01",
"version" : "6.1.1",
"http_address" : "127.0.0.1:9600",
"id" : "4f8524dc-38ac-4f3e-9563-bf965573761d",
"name" : "UAT-LOGSTASH-01",
"events" : {
"in" : 5000,
"filtered" : 5000,
"out" : 5000,
"duration_in_millis" : 9493,
"queue_push_duration_in_millis" : 824
}
}

previous value ==> "duration_in_millis" : 4284,
next value ===> "duration_in_millis" : 9493, (added with 4234 or new value?)

appreciate!!

[guyboertje]

Sure, I will try.

My understanding is that the duration_in_millis is the total time taken to process events in|filtered|out.

5000 / 9493 = 0.527 kEps or 527 events per second.

queue_push_duration_in_millis is the accumulative time the input are waiting to push events into the queue.

824 / 5000 = 0.185 ms or 185 microseconds on average. A high value here indicates a form of back-pressure where the outputs are slower than the inputs. Your value seems low, indicating that the outputs are keeping pace with the input cadence.

[shwesinhan (ssh)]

understand.
thank you @guyboertje