We have a Packetbeat 7.15.0 on Ubuntu 20.04 directly connected to a Debian 10 with Elasticsearch 7.14.1.
Packetbeat is running on a a DNS server with an average of 4000 queries/s. This is the configuration file:
packetbeat.interfaces.device: eno1
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.protocols:
- type: dns
ports: [53]
output.elasticsearch:
hosts: ["192.168.255.250:9200"]
processors:
- drop_fields:
fields:
[...]
monitoring.enabled: true
logging.to_files: true
logging.files:
path: /var/log/packetbeat
name: packetbeat
Packetbeat seems to drop (or to not emit) many events to Elasticsearch. These are the metrics:
If we enable debug (filtered on publisher), we can see almost instantly the log full of:
2021-09-27T11:37:36.485+0200 DEBUG [publisher] memqueue/produce.go:150 Dropping event, queue is blocked (seq=0)
Seq number is always 0. No matter what, it seems that we cannot have more than ~900 new event published to Elasticsearch.
We have tried to change many options but the one that seems to have effects is queue.mem.events. If we set to 102400 and restart Packetbeat we have a ~20 seconds of:
2021-09-27T11:37:23.258+0200 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [300: 0, 50]
2021-09-27T11:37:23.258+0200 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-09-27T11:37:23.258+0200 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
After ~20 seconds (~350 ack received) we start to see again in the log:
2021-09-27T11:37:36.485+0200 DEBUG [publisher] memqueue/produce.go:150 Dropping event, queue is blocked (seq=0)
2021-09-27T11:37:36.485+0200 DEBUG [publisher] pipeline/client.go:241 Pipeline client receives callback 'onDroppedOnPublish' for event: {...}
If we set queue.mem.events = 204800 the ack are ~700 and then it starts again with the error about queue is blocked.
If we set queue.mem.events = 409600 the ack are ~1500 and this is the situation:
New events = queued events but still emitted events (and acknowledged events) are much << than new events.
Where could be the problem? How can I debug such situation?
On a similar configuration (DNS server with Packetbeat + dedicated Elasticsearch server) we have this as expected:
All new events are emitted, acknowledged and queued.