Packet beat eats up 3GB of ram when running for a while

Elastic Stack Beats
1

Hi,

i've got packetbeat installed on a server and it sends its measurements to an elastic cluster (4 nodes, 12GB ram).
After a while (and on one particular server) packetbeat consumes 2-3GB RAM of memory and it makes the system unresponsive. The usual memory consuption for all the other servers is ~50-100MB.

I've enabled the logging and found that there is a huge amount of those lines in there.
Im not sure if those two issues are related (huge RAM consumption + the logs bellow), but that only happens on one out of ~30 nodes that have the same setup.

016-04-25T14:16:37+01:00 INFO Bulk item insert failed (i=40, status=429): {"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@204d577 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@470d82ed[Running, pool size = 8, active threads = 8, queued tasks = 50, completed tasks = 127309010]]"}
2

which packetbeat version have you running?

3 
packetbeat version 1.2.1 (amd64)

and my elasticsearch is

Version: 2.2.1, Build: d045fc2/2016-03-09T09:38:54Z, JVM: 1.8.0_65
4

what kind of traffic do you analyzer? can you share your packetbeat config? Memory usage depends on queue sizes and output configuration (e.g. number of ES hosts + workers).

Having these failures from one single node only is kinda funny, though. You still see traffic from this node? Status code 429 indicates elasticsearch queues being full (you're trying to index too many data).

A little dated, but check out this post.

5

I monitor for memcache, mysql and mongodb. My config seems pretty standard, but here it is

interfaces:
  device: any
protocols:
  memcache:
    # Configure the ports where to listen for memcache traffic. You can disable
    # the Memcache protocol by commenting out the list of ports.
    ports: [11211]
  mysql:
    # Configure the ports where to listen for MySQL traffic. You can disable
    # the MySQL protocol by commenting out the list of ports.
    ports: [3306]
  mongodb:
    # Configure the ports where to listen for MongoDB traffic. You can disable
    # the MongoDB protocol by commenting out the list of ports.
    ports: [27017]
output:

  ### Elasticsearch as output
  elasticsearch:
    hosts: [ "elastic01", "elastic02", "elastic03", "elastic04" ]

I get that failure in a couple of my ~30 servers but only one fails almost every time in this particular way.
Ill have a read through the post and see if i can find anything.

Cheers