In our company we've set up an elasticstack which has to handle about 130GB of logs (in indices) per day coming from 14 hosts (for now).
Yesterday I install x-pack monitoring on the filebeats but then I noticed that the CPU usage of the elasticsearch nodes (2) began to rise significantly.
It is a know issue that enabling filebeat monitoring needs a lot of CPU resources? If so, is there a fix for this?
If not, does anybody have any idea why the CPU usage would suddenly start to spike?
Monitoring should add some overhead to CPU usage, but it shouldn't be that significant. Can you try and reduce the interval at which the Filebeat monitoring is sending data and see if that makes a difference?
Also, can you show us some screenshots from the ES monitoring charts to show what kind of increase in CPU usage we're talking about. It sounds like something worth investigating further.
How can I reduce this interval, I didn't find a lot of info about x-pack monitoring for filebeat
Next to this I know that the CPU usage doubled and this is mainly 'User CPU time' but I'm not sure if the cause is X-pack since this is still an initial setup and changes are made daily.
On another note, have you considered not collecting X-Pack monitoring data into your production cluster but having it on its own dedicated Elasticsearch cluster? Say in the event that production has an issue and the monitoring data is unavailable which does not aid towards resolution.
That second off production cluster gives insight during these periods.
xpack.monitoring.collection.interval is the setting used by monitoring for Kibana and ES so it should be the same for Filebeat.
But if you change it, also change xpack.monitoring.min_interval_seconds option in kibana.yml to the same value.
The default interval is 10s. Change it to something that will fit your business needs, maybe 30s or 60s.
Allright, I'll give that a shot and see how it goes but I don't think solely x-pack is the problem
Now the CPU has calmed down a bit but is still higher than without monitoring
We didn't consider that because we are currently only using a basic license (looking for information before upgrading) so I just enabled the monitoring for some basic statistics
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.