Hi there,
I have recently deployed the ELK stack and beats to my servers, and for some of them I'm facing an issue with Packetbeat consuming 50-70% memory of the server
Beats: 7.8.0
Mongo: 4.0.6
This happens mostly during the night where I have some heavy jobs using mongodb, I tried to check for some improvements but didn't came across of much info, I only tried to set the queue parameter, but didn't change
I can see quite a lot of errors, just a few below
Aug 7 04:59:45 host1 packetbeat[9293]: 2020-08-07T04:59:45.437+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 04:59:48 host1 packetbeat[9293]: 2020-08-07T04:59:48.971+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 357535453 ((value=357535453))
Aug 7 04:59:49 host1 packetbeat[9293]: 2020-08-07T04:59:49.133+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1761577883 ((value=-1761577883))
Aug 7 04:59:49 host1 packetbeat[9293]: 2020-08-07T04:59:49.161+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917898 ((value=-1012917898))
Aug 7 05:15:12 host1 packetbeat[9293]: 2020-08-07T05:15:12.885+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 05:15:13 host1 packetbeat[9293]: 2020-08-07T05:15:13.996+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 151023215 ((value=151023215))
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.559+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.719+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917894 ((value=-1012917894))
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.839+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917918 ((value=-1012917918))
Aug 7 05:27:57 host1 packetbeat[9293]: 2020-08-07T05:27:57.632+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 10027064 ((value=10027064))
Aug 7 05:27:57 host1 packetbeat[9293]: 2020-08-07T05:27:57.655+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 10027057 ((value=10027057))
Aug 7 05:29:12 host1 packetbeat[9293]: 2020-08-07T05:29:12.787+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 33554463 ((value=33554463))
Aug 7 05:30:08 host1 packetbeat[9293]: 2020-08-07T05:30:08.530+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 687865856 ((value=687865856))
Aug 7 05:31:43 host1 packetbeat[9293]: 2020-08-07T05:31:43.983+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 808662837 ((value=808662837))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.419+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 909325106 ((value=909325106))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.455+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 859320373 ((value=859320373))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.457+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 876099890 ((value=876099890))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.499+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1633892096 ((value=1633892096))
Aug 7 08:09:16 host2 packetbeat[19533]: 2020-08-07T08:09:16.406+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:09:43 host2 packetbeat[19533]: 2020-08-07T08:09:43.743+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:09:43 host2 packetbeat[19533]: 2020-08-07T08:09:43.799+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 8257587 ((value=8257587))
Aug 7 08:10:08 host2 packetbeat[19533]: 2020-08-07T08:10:08.560+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:10:33 host2 packetbeat[19533]: 2020-08-07T08:10:33.337+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:10:58 host2 packetbeat[19533]: 2020-08-07T08:10:58.902+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:15:14 host2 packetbeat[19533]: 2020-08-07T08:15:14.215+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1212969300 ((value=1212969300))
Aug 7 08:16:08 host2 packetbeat[19533]: 2020-08-07T08:16:08.071+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:16:08 host2 packetbeat[19533]: 2020-08-07T08:16:08.072+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 8323128 ((value=8323128))
Aug 7 08:16:32 host2 packetbeat[19533]: 2020-08-07T08:16:32.412+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:16:38 host2 packetbeat[19533]: 2020-08-07T08:16:38.159+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 9961526 ((value=9961526))
Aug 7 08:17:02 host2 packetbeat[19533]: 2020-08-07T08:17:02.893+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:17:28 host2 packetbeat[19533]: 2020-08-07T08:17:28.012+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:26:12 host2 packetbeat[19533]: 2020-08-07T08:26:12.864+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:36:44 host2 packetbeat[19533]: 2020-08-07T08:36:44.881+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1769370213 ((value=1769370213))
Aug 7 08:36:44 host2 packetbeat[19533]: 2020-08-07T08:36:44.881+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 761753193 ((value=761753193))
Aug 7 08:36:58 host2 packetbeat[19533]: 2020-08-07T08:36:58.748+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 9961582 ((value=9961582))
Is this a defect on packetbeat that somehow is not "aware" of certain operations in mongo?
Also, I could understand maybe a higher usage during some spikes in the traffic but the memory is not released until the agent is restarted.
Does anyone has an idea on how to improve this situation? For now I left the agents stopped to not cause any performance issue on the servers.
packetbeat.yml
l
packetbeat.interfaces.device: "eth0"
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.ignore_outgoing: true
packetbeat.flows:
timeout: 30s
period: 300s
packetbeat.protocols:
-
type: dhcpv4
ports: [67, 68]
-
type: dns
ports: [53]
-
type: http
ports: [80, 8080, 8000, 5000, 8002]
-
type: mysql
ports: [3306,3307]
-
type: mongodb
ports: [27017]
-
type: nfs
ports: [2049]
-
type: tls
ports:- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
setup.template.settings:
index.number_of_shards:
hosts: ["host1:9200", "host2:9200"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
queue:
mem:
events: 4096
flush.min_events: 2048
flush.timeout: 1s
Thanks in advance.