Redis slowlog monitoring with ECK (Elastic on Kubernetes)?

[fzyzcjy]

Hi thanks for the wonderful elastic stack! I am trying to monitor the SLOWLOG of Redis in elastic, and I am on Kubernetes (using ECK). The doc says Redis filebeat for slowlog needs to specify a list of hosts. However, in k8s what hosts should I write down?

Say I have 10 nodes and 30 redis master-replica pairs. If I simply write down all 30 redis uris, then EACH of the 10 nodes will try to collect data. That is a disaster - 10x redundancy!

Thanks!

[jsoriano (Jaime Soriano)]

Hey @fzyzcjy, welcome to discuss

How are you configuring Filebeat? With autodiscover you can make Filebeat to discover automatically your running redis containers and add an only configuration for them, so you don't need to manually add additional entries for each one of your hosts. As you said, having to configure the list of hosts on all filebeat would have bad consequences, apart of being quite cumbersome

With autodiscover It would be something like this (not tested, but you can get the idea):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log
              slowlog:
                input:
                  type: redis
                  hosts: ["${data.host}:6379"]

This will start a redis module for each container with a image that contains "redis" on its name. The configuration includes some variables that will be replaced, in this case with the container id, and with its IP respectively.

You can read more about autodiscover here: https://www.elastic.co/guide/en/beats/filebeat/7.9/configuration-autodiscover.html

[fzyzcjy]

Thanks very much! Now I see how to implement it However I do not know how to safely add password to it? For example, I can put password in annotations and let elastic read it. But annotations are very unsafe to put password. Can I read k8s Secret from elastic? Thanks!

[jsoriano (Jaime Soriano)]

Take a look to this recent blogpost about managing credentials: https://www.elastic.co/blog/securely-manage-credentials-monitoring-kubernetes-workloads-autodiscovery

[fzyzcjy]

Thanks very much!

[fzyzcjy]

Hi I tried that but still failed Could you please provide some precious advice? Thanks! ---> Redis slowlog module + Kubernetes secrets => error :/