Metricbeat kibana no stable

[gerardgorrion]

We are triying to deploy metricbeat to monitor kibana in eck with stack monitoring. The problem is, the monitoring is not working fine:

Kibana stack monitoring:

Last 15 min of a single kibana node (continous monitoring is no working):

Example of self monitoring kibana node running ok:

Metricbeat:

elastic/cloud-on-k8s/blob/2.0/config/recipes/beats/stack_monitoring.yaml

Metricbeat config modules:

            templates:
              - condition:
                  contains:
                    kubernetes.labels.app: elasticsearch
                config:
                  - module: elasticsearch
                    metricsets:
                      - ccr
                      - cluster_stats
                      - enrich
                      - index
                      - index_recovery
                      - index_summary
                      - ml_job
                      - node_stats
                      - shard
                    period: 10s
                    hosts: "https://${data.host}:${data.ports.https}"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true
              - condition:
                  contains:
                    kubernetes.labels.app: elasticsearch
                config:
                  - module: kibana
                    metricsets:
                      - stats
                    period: 10s
                    hosts: "http://kibana-kb-http:5601"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true

Versions:
eck operator: 1.9
elasticseach and kibana: 7.16.2
metricbeat: 7.16.2

Thanks!

[gerardgorrion]

Here more tests:

Monitoring start: 08:30 with 2 kibana pod replicas.
At 08:31 put 1 replicas of kibana
At 08:38 put 3 replicas of kibana

Seem like when there're more than a replica, the monitoring is not working fine.

Some idea? Regards

[gerardgorrion]

Some idea if this metric is correctly? we think that can be some missconfig...

[Tetiana_Kravchenko]

Hi @gerardgorrion !

Sorry for the late reply.
The first thing I've noticed in your configuration is that you are using the same conditions for both Elasticsearch and kibana:

- condition:
    contains:
    kubernetes.labels.app: elasticsearch

do you see any issues with Elasticsearch metrics/dashboards?
could you please try to use more explicit conditions? and check if it helps somehow: you could use kubernetes.container.name for example.