Redis slowlog module + Kubernetes secrets => error :/

Hi thanks for the beautiful elastic stack! I am using Redis slowlog module + k8s secrets but then it fails.

EDIT: My guess - maybe the var.password do not understand k8s secrets? (just a naive guess)

The yaml

# ref: https://github.com/elastic/cloud-on-k8s/blob/master/config/recipes/beats/filebeat_autodiscover.yaml
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: {{ include "elastic.name.beat.filebeat" . }}
  labels: {{- include "elastic.labels" . | nindent 4 }}
spec:
  type: filebeat
  version: {{ .Values.filebeat.version }}
  image: "elastic/filebeat:{{ .Values.filebeat.version }}" 
  elasticsearchRef:
    name: {{ include "elastic.name.elasticsearch" . }}
  kibanaRef:
    name: {{ include "elastic.name.kibana" . }}
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            host: ${HOSTNAME}
            include_annotations:
              - tom_filebeat_mode
            hints:
              enabled: true
              default_config:
                type: container
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
            templates:
              {{- range $k, $v := .Values.filebeat.redis.modeSentinel }}
              - condition:
                  equals:
                    kubernetes.annotations.tom_filebeat_mode: {{ $v.release }}
                config:
                  # https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-redis.html
                  # and https://discuss.elastic.co/t/redis-slowlog-monitoring-with-eck-elastic-on-kubernetes/246377/2
                  - module: redis
                    log:
                      enabled: true
                      var.paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
                    slowlog:
                      enabled: {{ $.Values.filebeat.redis.enableSlowlog }}
                      {{- if $.Values.filebeat.redis.enableSlowlog }}
                      var.hosts: ["${data.host}:${data.kubernetes.annotations.redis_port}"]
                      # https://www.elastic.co/blog/securely-manage-credentials-monitoring-kubernetes-workloads-autodiscovery
                      var.password: "this_is_a_password" # FIRST
{{/*                      var.password: "${kubernetes.{{ $.Release.Namespace }}.{{ $v.release }}.redis-password}"*/}} # SECOND
                      {{- end }}
              {{- end }}
...

and the "template" part looks like the following (copied from the yaml of CRD in running k8s)

...
- condition:
                  equals:
                    kubernetes.annotations.tom_filebeat_mode: tom-redis-custom-code
                config:
                  - log:
                      enabled: true
                      var.paths:
                        - >-
                          /var/log/containers/*${data.kubernetes.container.id}.log
                    module: redis
                    slowlog:
                      enabled: true
                      var.hosts:
                        - '${data.host}:${data.kubernetes.annotations.redis_port}'
                      var.password: >-
                        ${kubernetes.default.tom-redis-custom-code.redis-password}
...

The related secret:

kind: Secret
apiVersion: v1
metadata:
  name: tom-redis-custom-code
  namespace: default
  selfLink: /api/v1/namespaces/default/secrets/tom-redis-custom-code
  uid: 4d20fa87-ff96-48a2-9abf-3ad0fb93323a
  resourceVersion: '79757'
  creationTimestamp: '2020-08-28T22:49:27Z'
  labels:
    app.kubernetes.io/instance: tom-redis-custom-code
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: tom-redis
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: tom-redis-0.1.0
  annotations:
    meta.helm.sh/release-name: tom-redis-custom-code
    meta.helm.sh/release-namespace: default
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2020-08-28T22:49:27Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          .: {}
          'f:redis-password': {}
        'f:metadata':
          'f:annotations':
            .: {}
            'f:meta.helm.sh/release-name': {}
            'f:meta.helm.sh/release-namespace': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/version': {}
            'f:helm.sh/chart': {}
        'f:type': {}
data:
  redis-password: dGhpc19pc19hX3Bhc3N3b3Jk
type: Opaque

In this case, when I manually create a (fake) slow command using redis-cli -h 172.18.0.14 -p 6379 -a this_is_a_password DEBUG SLEEP 1, it can report to elasticsearch and I can see the log. However, at the same time, there are a lot of 2020-08-28T23:57:41.427Z ERROR harvester/registry.go:98 Error running input: error receiving slowlog data: dial tcp 127.0.0.1:6379: connect: connection refused errors in Filebeat logs.

If I comment the # FIRST line and uncommend the # SECOND line, then everything does not work. No slow commands will be reported to elasticsearch and I cannot see it using kibana discovery. However, there is no more error in Filebeat log output!

I would appreciate for any suggestions. Thanks!

p.s. Is there a general methodology to debug such cases? e.g. enabling some logging or debugging?

Hey @fzyzcjy,

What version of filebeat are you using? Kubernetes keystore is only available since 7.9.0.

If you are using 7.9.0, do you see any other error in logs?

1 Like

Hi @jsoriano ,

Thanks for the reply! I am using 7.9.0 for everything. I do not see other errors.

By the way, is there some debug tools like: "List all variables that I can plug into some field in the yaml". If having this, I can simply show all variables and see what happens (e.g. the variable disappears)

There is nothing like that :confused:

@jsoriano So what should I do now :frowning: Thanks!

Hi @fzyzcjy!

That's great that you are -trying- to make use if the new Keystore!
Could you run Filebeat in debug mode (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html) and check for errors? I would firstly suspect that Filebeat has no access to k8s secrets due to some missing RBACs :thinking:, but let's see what logs can provide.

C.

I will have a try later :slight_smile: