Metric beat is not pushing data to elastic search - I am using metric beat to monitor host and dockers

Elastic Stack Beats
1

This is for Host

metricbeat.modules:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
#-------------------------- Elasticsearch output ------------------------------                                                                                                                      
output.elasticsearch:                                                                                                                                                                                  
username: "metricbeat_anonymous_user"                                                                                                                                                                
#password: ""                                                                                                                                                                                        
hosts: ["${HOST_ELASTICSEARCH}"]                                                                                                                                                                                                                                                                                                                                                                        
setup.kibana:                                                                                                                                                                                          
host: "${HOST_KIBANA}"                                                                                                                                                                                                                                                                                                                                                                                  
#============================== Dashboards =====================================                                                                                                                     
# These settings control loading the sample dashboards to the Kibana index. Loading                                                                                                                  
# the dashboards is disabled by default and can be enabled either by setting the                                                                                                                     
# options here, or by using the `-setup` CLI flag.                                                                                                                                                   
setup.dashboards.enabled: true                                                                                                                                                                                                                                                                                                                                                                            
logging.level: warning                                                                                                                                                                               
logging.to_files: true                                                                                                                                                                               
logging.to_syslog: false                                                                                                                                                                             
logging.files:                                                                                                                                                                                         
path: /var/log/metricbeat                                                                                                                                                                            
name: metricbeat.log                                                                                                                                                                                 
keepfiles: 2                                                                                                                                                                                         
permissions: 0644
2

This is for containers -

metricbeat.modules: - module: system metricsets: ["cpu", "load", "filesystem", "fsstat", "memory", "network", "process", "core", "diskio", "socket"] period: 5s enabled: true processes: ['.*'] cpu.metrics: ["percentages"] core.metrics: ["percentages"] #------------------------------- Docker Module ------------------------------- - module: docker metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"] hosts: ["unix:///var/run/docker.sock"] enabled: true period: 5s #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: username: "metricbeat_anonymous_user" #password: "" hosts: ["{HOST_ELASTICSEARCH}"] setup.kibana: host: "{HOST_KIBANA}" setup.dashboards.enabled: true logging.level: warning logging.to_files: true logging.to_syslog: false logging.files: -----

3

My docker-compose file -

version: "2.2" services: # Used as the Metricbeat output # Of course, we could create a cluster with more nodes elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4 container_name: metricbeat-elasticsearch memswap_limit: 0 mem_limit: 2g cap_add: - IPC_LOCK environment: - cluster.name=docker-cluster - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms1g -Xmx1g" ulimits: memlock: soft: -1 hard: -1 volumes: - ./config/usr/share/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml ports: - "9222:9200" networks: - metricbeat

4

Continuation .... docker-compose

Used to analyse data sent to elasticsearch from Metricbeat. # Metricbeat directly creates Dashbords into Kibana kibana: image: docker.elastic.co/kibana/kibana:6.5.4 container_name: metricbeat-kibana volumes: - ./config/opt/kibana/config/kibana.yml:/opt/kibana/config/kibana.yml depends_on: - elasticsearch ports: - "5666:5601" networks: - metricbeat

5

continuation docker compose for metric beat -

metricbeat: build: ./docker/metricbeat container_name: metricbeat-metricbeat-services command: -e environment: - "WAIT_FOR_HOSTS=elasticsearch:9200 kibana:5601" - "HOST_ELASTICSEARCH=elasticsearch:9200" - "HOST_KIBANA=kibana:5601" depends_on: - elasticsearch networks: - metricbeat

6

for metricbeat -host portion

metricbeat-host: build: context: ./docker/metricbeat args: - METRICBEAT_FILE=metricbeat-host.yml container_name: metricbeat-metricbeat-host command: -system.hostfs=/hostfs volumes: - /proc:/hostfs/proc:ro - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro - /:/hostfs:ro - /var/run/docker.sock:/var/run/docker.sock environment: - "WAIT_FOR_HOSTS=elasticsearch:9222 kibana:5666" - "HOST_ELASTICSEARCH=elasticsearch:9222" - "HOST_KIBANA=kibana:5666" extra_hosts: - "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,... networks: metricbeat: external: name: metricbeat

7

d
d.png1600×900 79.4 KB

8

This Waiting for Cluster... message continuously coming. Using ELK -6.5.4 and Metricbeat 6.5.4

9

Any suggestions.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.