Some problem of logstash elapsed filter


#1

I use elapsed filter to calculate events duration, but it has something wrong. The elapsed_time got negative number in my case.

config below:

Can someone help me?
Thanks.


(Lewis Barclay) #2

Your start and end events coming in very quickly, 100ms from each other is that correct?


#3

Thanks for your reply first. I got a search result for elpased_time between 0 to 1, like below pic.

Sometime elapsed get work when events between 100ms. But it maybe duration of events too fast to got this problem. And, I want to let it become normally to work correct. Hope for more discuss. Thanks.


(Lewis Barclay) #4

Correct I think possibly the events are too fast


#5

Have other solution´╝čI want to use elapsed to do performance report.


(Lewis Barclay) #6

I am unsure, perhaps raise a bug on github with the logstash team?


(Christian Dahlqvist) #7

How many worker threads do you have configured for Logstash?


#8

It has default setting, but I think it have 4 threads.


(Christian Dahlqvist) #9

If I recall correctly, one of the major drawbacks with the elapsed and aggregate filters is that all related events have to pass through the same processing thread, thereby limiting Logstash to a single worker thread and instance. This naturally severely limits throughput and scales very badly. Try setting the number of worker threads to 1 to see if it resolves the accuracy issue.

A more scalable approach may be to have a periodic batch job that post-process events once they are in Elasticsearch, but although this is likely to scale better it does introduce a delay.


#10

There are some wrong elapsed time after change the thread to 1 worker. Changing thread maybe let it work, but still have some wrong elapsed.


Ooes pipeline.batch.size and pipeline.batch.delay have to change?


(Christian Dahlqvist) #11

What does the underlying events that contributed to an incorrect elapsed time look like if you view them in Kibana?


#12

I take a few events picture. Its tag seem like missing start_tag.

Do my elapsed code wrong in logstash.conf? Doesn't have to use else if?


#13

Hello, who can tell me where is going wrong?