Hi,
I'm using the elk suite (all in 6.8.6).
I've installed metricbeat on all my hosts (more than 250).
Config is identical on all hosts:
#========================== Modules configuration ============================
metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
metricsets:- cpu # CPU usage
- load # CPU load averages
- memory # Memory usage
- network # Network IO
- process # Per process metrics
- process_summary # Process summary
- uptime # System Uptime
#- core # Per CPU core usage
#- diskio # Disk IO - filesystem # File system usage for each mountpoint
- fsstat # File system summary metrics
#- raid # Raid
#- socket # Sockets and connection info (linux only)
enabled: true
period: 60s
processes: ['.*']
...
Data is sent to a Load Balancer which have 2 Logstash servers as real servers.
Logstash deamon enrich MetricBeat data with few fields using mapping files (based on host names).
Data is then sent to ElasticSearch to index with the following specification:
index => "metricbeat-%{[metricset][name]}-%{+YYYY.MM.dd}"
I can see the data in ElasticSearch. As an example, the index are:
metricbeat-cpu-*
metricbeat-memory-*
...
In Kibana, in Infrastructure UI, I can see few hosts... But not all of them...
I mean, that when I restart all my metricbeat, I can see the host appearing in the infra, and few minutes later, they start to disappear.
Looking to ElasticSearc, I've gaps up to 30 / 60 mn while I'm not receiving anything, and without doing anything, data starts to be inserted again, and it stops...and again, and again...
I read lots of docs, I did not find anything. The cluster ElasticSearch has 13 VMs with 128GB RAM and 12vCPUs each (disk space is 6.5 TB used of 12.0 TB total)
Do you have any idea of what could be the problem ?
Regards