Some of my agents are sending 1GB of data every hour. They are all laptops and desktops with a basic setup.
Would they do that if the elasticsearch destination they are sending data to is down?
Just to add some more information.
I found out that my cluster was no longer writing any new data. Once I rebooted the cluster, and confirmed new data, bandwidth from the clients dropped significantly.
I'm hoping someone might have some insight about what agent settings I could modify to prevent this from happening?
It seems very odd that something malfunctioning on the cluster would cause so much traffic from the agents.
Hello @sourcreamnormanbates
Is it possible that the enqueued data hit your configured maximum queue size? Once the queue is full they stop reading new data. If the events are ephemeral or log files that rotate this can lead to data loss.
For events from Elastic Defend there is a disk queue, for other integrations the events are queued in memory.
Yes, that is at least partially what has happened.
During busy hours of the day, I'm often seeing errors in the Kibana log.
"statusCode":429,"error":"Too Many Requests","message":"[es_rejected_execution_exception......
....
Running, pool size = 8, active threads = 8, queued tasks = 10172, completed tasks = 236470315
.......
I have verified in the elasticsearch.yml the default queue size of 10000 is configured.
I believe the issue is the virtual host is running out of CPU capacity. The host is alerting on CPU usage consistently being over 90%.
I think I need to get more resources available from the server team before I troubleshoot this any further.
Yes, I agree. It seems likely the CPU is the bottleneck. I would try giving the host more resources.
Come back if there's anything else I can help.
It may be some time before I have more resources available.
I can open another separate ticket if you'd like, but I'm wondering if I should look at optimizing the integrations deployed to my agents.
I'm running the Elastic Defend, Windows, and System integrations.
With Elastic Defend collecting events like DNS, File, Network, Process, Registry, and Security, do I really need to also have both the Windows and System integrations also enabled?
EDIT: I should have thought of this a couple more minutes before posting. Many of the SIEM rules require those two integrations, so I'll just have to wait for the resources.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.