Setup Stack Monitoring with Metricbeat and Docker-compose

[damasu (Jesse Quinn)]

I am wondering how to correctly configure stack monitoring in kibana when used with an elastic stack based on docker-compose.

For instance, on the same host machine as docker-compose i have installed metricbeat to monitor the containers es01, es02, es03:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

metricbeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition.contains:
            docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "CHANGME"
        - condition.contains:
            docker.container.name: es01
          config:
            - module: elasticsearch
              metricsets: ["node", "node_stats"]
              hosts: "https://${data.host}:9200"
              username: "elastic"
              password: "CHANGME"
              ssl.certificate_authorities: ["ca.crt"]
              xpack.enabled: false
              period: 10s
        - condition.contains:
            docker.container.name: es02
          config:
            - module: elasticsearch
              metricsets: ["node", "node_stats"]
              hosts: "https://${data.host}:9200"
              username: "elastic"
              password: "CHANGME"
              ssl.certificate_authorities: ["ca.crt"]
              xpack.enabled: false
              period: 10s
        - condition.contains:
            docker.container.name: es03
          config:
            - module: elasticsearch
              metricsets: ["node", "node_stats"]
              hosts: "https://${data.host}:9200"
              username: "elastic"
              password: "CHANGME"
              ssl.certificate_authorities: ["ca.crt"]
              xpack.enabled: false
              period: 10s
        - condition.contains:
            docker.container.name: kibana
          config:
            - module: kibana
              metricsets: ["status"]
              hosts: "https://${data.host}:5601"
              username: "elastic"
              password: "CHANGME"
              period: 10s
              basepath: ""
              ssl.verification_mode: "none"
        - condition.contains:
            docker.container.name: logstash
          config:
            - module: logstash
              metricsets: ["node", "node_stat"]
              hosts: "${data.host}:9600"
              period: 10s

#setup.template.settings:
#  index.number_of_shards: 1
#  index.codec: best_compression

#setup.template.name: "metricbeat-"
#setup.template.pattern: "metricbeat-*"

#setup.dashboards.enabled: true

#setup.kibana:
#  host: "localhost:5601"
#  protocol: "https"
#  username: "elastic"
#  password: "CHANGME"

#output.elasticsearch:
#  enabled: true
#  hosts: ["localhost:9200"]
#  protocol: "https"
#  username: "elastic"
#  password: "CHANGME"
#  ssl.enabled: true
#  ssl.certificate_authorities: ["ca.crt"]
#  ssl.certificate: "es01.crt"
#  ssl.key: "es01.key"

output.redis:
  hosts: ["localhost"]
  password: "CHANGME"
  key: "metricbeat"
  db: 0
  timeout: 5
  data_type: "list"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I can see in the metricbeat-* index under discovery in kibana that i am receiving data. However, Kibana stack monitoring continues to show the button "monitor with metricbeat". What exactly do i need to do so that kibana stack monitoring sees the data collected in the metricbeat index?

I am using static ips in docker-compose as an example

networks:
  elastic:
    ipam:
      config:
        - subnet: 172.20.0.0/24

...

    networks:
      elastic:
        ipv4_address: 172.20.0.8

...

i can also curl --cacert ca.crt -u elastic:CHANGEME 'https://172.20.0.8:9200' and i get the version etc response as expected.

If i set xpack.enabled: true I still have issues as i am using output.redis. Is it possible to use this functionality with the redis output?

by adding

monitoring.enabled: true
#monitoring.cluster_uuid:
monitoring.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "https"
  username: "elastic"
  password: "CHANGME"
  ssl.enabled: true
  ssl.certificate_authorities: ["ca.crt"]
  ssl.certificate: "es01.crt"
  ssl.key: "es01.key"

i can now see the stack monitoring; however, it shows no nodes for logstash or elasticsearch.

I have been able to setup it for the most part. However, those remote metricbeats and filebeats that use output.redis do not populate the correct index. What exactly would be needed to the logstash.conf to output correct the xpack monitoring feature?

input {
  beats {
    port => 5044
  }

  redis {
    host => "redis"
    password => "CHANGME"
    key => "filebeat"
    data_type => "list"
  }

  redis {
    host => "redis"
    password => "CHANGME"
    key => "metricbeat"
    data_type => "list"
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://es01:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}"
      cacert => "/usr/share/elasticsearch/config/certificates/ca/ca.crt"
      user => "logstash_writer"
      password => "CHANGME"
    }
  } else {
    elasticsearch {
      hosts => "https://es01:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      cacert => "/usr/share/elasticsearch/config/certificates/ca/ca.crt"
      user => "logstash_writer"
      password => "CHANGME"
    }
  }
}

by adding monitoring.cluster_uuid: some uuid resolves the issue with output.redis. I had to use output.elasticsearch for kibana, logstash, and cluster of elasticsearches to work. but all filebeats/metricbeats remotely can use redis with the monitoring.cluster_uuid set.

i think everything is solved.