I've been characterizing a problem with Packetbeat, configured to directly output to Elasticsearch. I've got the YML configured to use the memory internal queue (not the file spool queue).
The problem is, when I burst 250 events, all attempting to be published within 500 milliseconds or so, there's times when not all the 250 events show up in Elasticsearch. I've turned on enough debugging to see this in the packetbeat log (set to debug level):
2019-11-24T15:56:44.069Z DEBUG [publisher] memqueue/produce.go:155 Dropping event, queue is blocked (seq=138)
I see this log the same number of times as events I'm missing in Elasticsearch. For some bursts there's no loss, some have significant loss, it varies so it seems to be some sort of race condition or performance issue.
What I'd like to know most is why would this memory queue be blocked? Here's the important sections of the packetbeat.yml file I'm using related to this:
I've tried various combinations of settings for queue:mem:flush.min_events and timeouts, nothing seems to help.
Explanation on why this blocking is occurring would be most welcome!