I've configured packetbeat on a machine with plenty resources (72GB RAM, 24 cores Intel Xeon X5670), and its listening on two 10gb interfaces, that combined are reaching about 1gbps during peak hours. On this same machine I'm running suricata. It and Packetbeat listening on the same interfaces. Suricata is currently using about 700% of CPU power and 5GB of RAM. Packetbeat tops at about 300% CPU and 4GB RAM.
But I'm seeing some events acumulating and probably some packet loses of packetbeat, as can been seen bellow:
On the machine I can see that it hangs from time to time, by following the journal (the machine is a RH 7.9 fully patched), as seen bellow.
Jan 25 10:42:39 XXXXX packetbeat: 2021-01-25T10:42:39.135-0300 ERROR [pgsql] pgsql/parse.go:544 Pgsql invalid column_length=4294967295, buffer_length=62, i=3 Jan 25 10:45:17 XXXXX packetbeat: 2021-01-25T10:45:17.673-0300 ERROR [pgsql] pgsql/parse.go:544 Pgsql invalid column_length=4294967295, buffer_length=172, i=8
For what I could see that vast majority of events come when the journal "stops". The packets never stop comming on the interfaces.
I've already tried stopping suricata, disabling flows, changing flow period (using the default 30s), changing the internal queue size, etc. But I'm always ending with this problem and a max of 10k events per second on this machine.
My ES cluster is very lightlly loaded (we have other sources arriving on it, and I've seen it reach about 70k primary events per second), I don't have any aditional pipelines running on these events. We have 3 master nodes, 4 ingest nodes and 10 data nodes.
Can you suggest what the problem may be or configuration that I can try? Thanks!
My packetbeat configuration is this:
packetbeat.flows: enabled: true period: -1s timeout: 30s packetbeat.interfaces.device: any packetbeat.interfaces.type: af_packet packetbeat.interfaces.buffer_size_mb: 100 packetbeat.interfaces.with_vlan: true max_procs: 256 packetbeat.ignore_outgoing: true packetbeat.interfaces.auto_promisc_mode: true packetbeat.interfaces.snaplen: 1514 packetbeat.protocols: - enabled: false type: icmp - ports: - 5672 type: amqp - ports: - 9042 type: cassandra - ports: - 67 - 68 type: dhcpv4 - ports: - 53 type: dns - ports: - 80 - 8080 - 8000 - 5000 - 8002 - 804 type: http - type: memcache - ports: - 3306 - 3307 type: mysql - ports: - 5432 type: pgsql - ports: - 6379 type: redis - ports: - 9090 type: thrift - ports: - 27017 type: mongodb - ports: - 2049 type: nfs - ports: - 443 - 993 - 995 - 5223 - 8443 - 8883 - 9243 type: tls processors: setup.ilm.overwrite: false setup.kibana: host: https://XXXXXXXXX:443 ssl.verification_mode: certificate setup.template.settings: index.number_of_shards: 5 ssl.certificate_authorities: /etc/packetbeat/certs/XXXXXXXXX.pem output: elasticsearch: bulk_max_size: 200 compression_level: 0 hosts: - https://XXXXXXXXX:9200 - https://XXXXXXXXX:9200 - https://XXXXXXXXX:9200 - https://XXXXXXXXX:9200 loadbalance: true password: XXXXXXXXX ssl.verification_mode: certificate username: XXXXXXXXX worker: 1 logging.level: "error" logging: files: rotateeverybytes: 10485760