Fleet server, connection limits and duplicate agents

[blinder (Björn Linder)]

Hi

We recently enrolled about 300 fleet clients to our On-Prem ECE cluster, enrollment seems to be working fine but duplicates( marked as offline) are spawning en mass. Data are being ingested from machines with double entries and i can unenroll the duplicates based on time last seen. But why is this happening?

In addition to that, about half of the clients are "offline" at any given time with the following error code in the logs

[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: status code: 429, fleet-server returned an error: MaxLimit, message: exceeded the max limit

I have followed the settings guide for the fleet server integration policy and adjusted it to accomodate 5000 client, with no change.

cache:

  num_counters: 10000    

  max_cost: 10485760     

server.limits:

   policy_throttle: 100ms

   checkin_limit:

     interval: 10ms      

     burst: 2500          

     max: 2601           

   artifact_limit:

     interval: 10ms      

     burst: 2500          

     max: 5000            

   ack_limit:

     interval: 8ms       

     burst: 2500          

     max: 5000            

   enroll_limit:

     interval: 40ms      

     burst: 250           

     max: 500             

server.runtime:

  gc_percent: 20

Any ideas?