# Redis slowlog module + Kubernetes secrets =\> error :/

**URL:** <https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 29, 2020, 12:00am UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827 "2020-08-29T00:00:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fzyzcjy/32/78628_2.png) [@fzyzcjy](https://discuss.elastic.co/u/fzyzcjy)\
**Post date:** [August 29, 2020, 12:00am UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/1 "2020-08-29T00:00:51Z")

</div>

Hi thanks for the beautiful elastic stack! I am using Redis slowlog module + k8s secrets but then it fails.

**EDIT:** My guess - maybe the `var.password` do not understand k8s secrets? (just a naive guess)

The yaml

```auto
# ref: https://github.com/elastic/cloud-on-k8s/blob/master/config/recipes/beats/filebeat_autodiscover.yaml
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: {{ include "elastic.name.beat.filebeat" . }}
  labels: {{- include "elastic.labels" . | nindent 4 }}
spec:
  type: filebeat
  version: {{ .Values.filebeat.version }}
  image: "elastic/filebeat:{{ .Values.filebeat.version }}" 
  elasticsearchRef:
    name: {{ include "elastic.name.elasticsearch" . }}
  kibanaRef:
    name: {{ include "elastic.name.kibana" . }}
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            host: ${HOSTNAME}
            include_annotations:
              - tom_filebeat_mode
            hints:
              enabled: true
              default_config:
                type: container
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
            templates:
              {{- range $k, $v := .Values.filebeat.redis.modeSentinel }}
              - condition:
                  equals:
                    kubernetes.annotations.tom_filebeat_mode: {{ $v.release }}
                config:
                  # https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-redis.html
                  # and https://discuss.elastic.co/t/redis-slowlog-monitoring-with-eck-elastic-on-kubernetes/246377/2
                  - module: redis
                    log:
                      enabled: true
                      var.paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
                    slowlog:
                      enabled: {{ $.Values.filebeat.redis.enableSlowlog }}
                      {{- if $.Values.filebeat.redis.enableSlowlog }}
                      var.hosts: ["${data.host}:${data.kubernetes.annotations.redis_port}"]
                      # https://www.elastic.co/blog/securely-manage-credentials-monitoring-kubernetes-workloads-autodiscovery
                      var.password: "this_is_a_password" # FIRST
{{/* var.password: "${kubernetes.{{ $.Release.Namespace }}.{{ $v.release }}.redis-password}"*/}} # SECOND
                      {{- end }}
              {{- end }}
...

```

and the "template" part looks like the following (copied from the yaml of CRD in running k8s)

```auto
...
- condition:
                  equals:
                    kubernetes.annotations.tom_filebeat_mode: tom-redis-custom-code
                config:
                  - log:
                      enabled: true
                      var.paths:
                        - >-
                          /var/log/containers/*${data.kubernetes.container.id}.log
                    module: redis
                    slowlog:
                      enabled: true
                      var.hosts:
                        - '${data.host}:${data.kubernetes.annotations.redis_port}'
                      var.password: >-
                        ${kubernetes.default.tom-redis-custom-code.redis-password}
...

```

The related secret:

```auto
kind: Secret
apiVersion: v1
metadata:
  name: tom-redis-custom-code
  namespace: default
  selfLink: /api/v1/namespaces/default/secrets/tom-redis-custom-code
  uid: 4d20fa87-ff96-48a2-9abf-3ad0fb93323a
  resourceVersion: '79757'
  creationTimestamp: '2020-08-28T22:49:27Z'
  labels:
    app.kubernetes.io/instance: tom-redis-custom-code
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: tom-redis
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: tom-redis-0.1.0
  annotations:
    meta.helm.sh/release-name: tom-redis-custom-code
    meta.helm.sh/release-namespace: default
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2020-08-28T22:49:27Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          .: {}
          'f:redis-password': {}
        'f:metadata':
          'f:annotations':
            .: {}
            'f:meta.helm.sh/release-name': {}
            'f:meta.helm.sh/release-namespace': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/version': {}
            'f:helm.sh/chart': {}
        'f:type': {}
data:
  redis-password: dGhpc19pc19hX3Bhc3N3b3Jk
type: Opaque

```

In this case, when I manually create a (fake) slow command using `redis-cli -h 172.18.0.14 -p 6379 -a this_is_a_password DEBUG SLEEP 1`, it can report to elasticsearch and I can see the log. However, at the same time, there are a lot of `2020-08-28T23:57:41.427Z	ERROR	harvester/registry.go:98	Error running input: error receiving slowlog data: dial tcp 127.0.0.1:6379: connect: connection refused` errors in Filebeat logs.

If I comment the `# FIRST` line and uncommend the `# SECOND` line, then everything does _not_ work. No slow commands will be reported to elasticsearch and I cannot see it using kibana discovery. However, _there is no **more** error in Filebeat log output!_

I would appreciate for any suggestions. Thanks!

p.s. Is there a general methodology to debug such cases? e.g. enabling some logging or debugging?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 30, 2020, 12:00pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/2 "2020-08-30T12:00:07Z")

</div>

Hey @fzyzcjy,

What version of filebeat are you using? Kubernetes keystore is only available since 7.9.0.

If you are using 7.9.0, do you see any other error in logs?

---

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fzyzcjy/32/78628_2.png) [@fzyzcjy](https://discuss.elastic.co/u/fzyzcjy)\
**Post date:** [August 30, 2020, 12:18pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/3 "2020-08-30T12:18:39Z")

</div>

Hi @jsoriano ,

Thanks for the reply! I am using 7.9.0 for everything. I do not see other errors.

By the way, is there some debug tools like: "List all variables that I can plug into some field in the yaml". If having this, I can simply show all variables and see what happens (e.g. the variable disappears)

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 30, 2020, 1:16pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/4 "2020-08-30T13:16:09Z")

</div>

> [@fzyzcjy](#):
>
> By the way, is there some debug tools like: "List all variables that I can plug into some field in the yaml". If having this, I can simply show all variables and see what happens (e.g. the variable disappears)

There is nothing like that 😕

---

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fzyzcjy/32/78628_2.png) [@fzyzcjy](https://discuss.elastic.co/u/fzyzcjy)\
**Post date:** [August 30, 2020, 1:59pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/5 "2020-08-30T13:59:51Z")

</div>

@jsoriano So what should I do now ☹ Thanks!

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [August 31, 2020, 9:23am UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/6 "2020-08-31T09:23:15Z")

</div>

Hi @fzyzcjy!

That's great that you are -trying- to make use if the new Keystore!  
Could you run Filebeat in debug mode ([https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html)) and check for errors? I would firstly suspect that Filebeat has no access to k8s secrets due to some missing RBACs 🤔, but let's see what logs can provide.

C.

---

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fzyzcjy/32/78628_2.png) [@fzyzcjy](https://discuss.elastic.co/u/fzyzcjy)\
**Post date:** [September 4, 2020, 1:32pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/7 "2020-09-04T13:32:40Z")

</div>

I will have a try later 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2020, 3:32pm UTC](https://discuss.elastic.co/t/redis-slowlog-module-kubernetes-secrets-error/246827/8 "2020-10-02T15:32:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.