Packetbeat memory usage

Hi there,

I have recently deployed the ELK stack and beats to my servers, and for some of them I'm facing an issue with Packetbeat consuming 50-70% memory of the server

Beats: 7.8.0
Mongo: 4.0.6

This happens mostly during the night where I have some heavy jobs using mongodb, I tried to check for some improvements but didn't came across of much info, I only tried to set the queue parameter, but didn't change

I can see quite a lot of errors, just a few below

Aug 7 04:59:45 host1 packetbeat[9293]: 2020-08-07T04:59:45.437+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 04:59:48 host1 packetbeat[9293]: 2020-08-07T04:59:48.971+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 357535453 ((value=357535453))
Aug 7 04:59:49 host1 packetbeat[9293]: 2020-08-07T04:59:49.133+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1761577883 ((value=-1761577883))
Aug 7 04:59:49 host1 packetbeat[9293]: 2020-08-07T04:59:49.161+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917898 ((value=-1012917898))
Aug 7 05:15:12 host1 packetbeat[9293]: 2020-08-07T05:15:12.885+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 05:15:13 host1 packetbeat[9293]: 2020-08-07T05:15:13.996+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 151023215 ((value=151023215))
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.559+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.719+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917894 ((value=-1012917894))
Aug 7 05:27:25 host1 packetbeat[9293]: 2020-08-07T05:27:25.839+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: -1012917918 ((value=-1012917918))
Aug 7 05:27:57 host1 packetbeat[9293]: 2020-08-07T05:27:57.632+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 10027064 ((value=10027064))
Aug 7 05:27:57 host1 packetbeat[9293]: 2020-08-07T05:27:57.655+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 10027057 ((value=10027057))
Aug 7 05:29:12 host1 packetbeat[9293]: 2020-08-07T05:29:12.787+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 33554463 ((value=33554463))
Aug 7 05:30:08 host1 packetbeat[9293]: 2020-08-07T05:30:08.530+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 687865856 ((value=687865856))
Aug 7 05:31:43 host1 packetbeat[9293]: 2020-08-07T05:31:43.983+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 808662837 ((value=808662837))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.419+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 909325106 ((value=909325106))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.455+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 859320373 ((value=859320373))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.457+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 876099890 ((value=876099890))
Aug 7 05:31:44 host1 packetbeat[9293]: 2020-08-07T05:31:44.499+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1633892096 ((value=1633892096))

Aug 7 08:09:16 host2 packetbeat[19533]: 2020-08-07T08:09:16.406+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:09:43 host2 packetbeat[19533]: 2020-08-07T08:09:43.743+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:09:43 host2 packetbeat[19533]: 2020-08-07T08:09:43.799+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 8257587 ((value=8257587))
Aug 7 08:10:08 host2 packetbeat[19533]: 2020-08-07T08:10:08.560+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:10:33 host2 packetbeat[19533]: 2020-08-07T08:10:33.337+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:10:58 host2 packetbeat[19533]: 2020-08-07T08:10:58.902+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:15:14 host2 packetbeat[19533]: 2020-08-07T08:15:14.215+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1212969300 ((value=1212969300))
Aug 7 08:16:08 host2 packetbeat[19533]: 2020-08-07T08:16:08.071+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:16:08 host2 packetbeat[19533]: 2020-08-07T08:16:08.072+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 8323128 ((value=8323128))
Aug 7 08:16:32 host2 packetbeat[19533]: 2020-08-07T08:16:32.412+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:16:38 host2 packetbeat[19533]: 2020-08-07T08:16:38.159+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 9961526 ((value=9961526))
Aug 7 08:17:02 host2 packetbeat[19533]: 2020-08-07T08:17:02.893+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:17:28 host2 packetbeat[19533]: 2020-08-07T08:17:28.012+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:26:12 host2 packetbeat[19533]: 2020-08-07T08:26:12.864+0200#011ERROR#011mongodb/mongodb_parser.go:148#011An error occurred while parsing OP_REPLY message: document out of bounds
Aug 7 08:36:44 host2 packetbeat[19533]: 2020-08-07T08:36:44.881+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 1769370213 ((value=1769370213))
Aug 7 08:36:44 host2 packetbeat[19533]: 2020-08-07T08:36:44.881+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 761753193 ((value=761753193))
Aug 7 08:36:58 host2 packetbeat[19533]: 2020-08-07T08:36:58.748+0200#011ERROR#011mongodb/mongodb_parser.go:68#011Unknown operation code: 9961582 ((value=9961582))

Is this a defect on packetbeat that somehow is not "aware" of certain operations in mongo?

Also, I could understand maybe a higher usage during some spikes in the traffic but the memory is not released until the agent is restarted.

Does anyone has an idea on how to improve this situation? For now I left the agents stopped to not cause any performance issue on the servers.

packetbeat.yml

l

packetbeat.interfaces.device: "eth0"
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.ignore_outgoing: true

packetbeat.flows:
timeout: 30s

period: 300s

packetbeat.protocols:

  • type: dhcpv4

    ports: [67, 68]

  • type: dns

    ports: [53]

  • type: http

    ports: [80, 8080, 8000, 5000, 8002]

  • type: mysql

    ports: [3306,3307]

  • type: mongodb

    ports: [27017]

  • type: nfs

    ports: [2049]

  • type: tls
    ports:

    • 443 # HTTPS
    • 993 # IMAPS
    • 995 # POP3S
    • 5223 # XMPP over SSL
    • 8443
    • 8883 # Secure MQTT
    • 9243 # Elasticsearch

setup.template.settings:
index.number_of_shards:

hosts: ["host1:9200", "host2:9200"]

processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~
  • add_docker_metadata: ~

queue:
mem:
events: 4096
flush.min_events: 2048
flush.timeout: 1s

Thanks in advance.

Any thoughts on this anybody?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.