Hi thanks for the beautiful elastic stack! I am using Redis slowlog module + k8s secrets but then it fails.
EDIT: My guess - maybe the var.password do not understand k8s secrets? (just a naive guess)
The yaml
# ref: https://github.com/elastic/cloud-on-k8s/blob/master/config/recipes/beats/filebeat_autodiscover.yaml
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: {{ include "elastic.name.beat.filebeat" . }}
labels: {{- include "elastic.labels" . | nindent 4 }}
spec:
type: filebeat
version: {{ .Values.filebeat.version }}
image: "elastic/filebeat:{{ .Values.filebeat.version }}"
elasticsearchRef:
name: {{ include "elastic.name.elasticsearch" . }}
kibanaRef:
name: {{ include "elastic.name.kibana" . }}
config:
filebeat:
autodiscover:
providers:
- type: kubernetes
host: ${HOSTNAME}
include_annotations:
- tom_filebeat_mode
hints:
enabled: true
default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
templates:
{{- range $k, $v := .Values.filebeat.redis.modeSentinel }}
- condition:
equals:
kubernetes.annotations.tom_filebeat_mode: {{ $v.release }}
config:
# https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-redis.html
# and https://discuss.elastic.co/t/redis-slowlog-monitoring-with-eck-elastic-on-kubernetes/246377/2
- module: redis
log:
enabled: true
var.paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
slowlog:
enabled: {{ $.Values.filebeat.redis.enableSlowlog }}
{{- if $.Values.filebeat.redis.enableSlowlog }}
var.hosts: ["${data.host}:${data.kubernetes.annotations.redis_port}"]
# https://www.elastic.co/blog/securely-manage-credentials-monitoring-kubernetes-workloads-autodiscovery
var.password: "this_is_a_password" # FIRST
{{/* var.password: "${kubernetes.{{ $.Release.Namespace }}.{{ $v.release }}.redis-password}"*/}} # SECOND
{{- end }}
{{- end }}
...
and the "template" part looks like the following (copied from the yaml of CRD in running k8s)
...
- condition:
equals:
kubernetes.annotations.tom_filebeat_mode: tom-redis-custom-code
config:
- log:
enabled: true
var.paths:
- >-
/var/log/containers/*${data.kubernetes.container.id}.log
module: redis
slowlog:
enabled: true
var.hosts:
- '${data.host}:${data.kubernetes.annotations.redis_port}'
var.password: >-
${kubernetes.default.tom-redis-custom-code.redis-password}
...
The related secret:
kind: Secret
apiVersion: v1
metadata:
name: tom-redis-custom-code
namespace: default
selfLink: /api/v1/namespaces/default/secrets/tom-redis-custom-code
uid: 4d20fa87-ff96-48a2-9abf-3ad0fb93323a
resourceVersion: '79757'
creationTimestamp: '2020-08-28T22:49:27Z'
labels:
app.kubernetes.io/instance: tom-redis-custom-code
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: tom-redis
app.kubernetes.io/version: 1.16.0
helm.sh/chart: tom-redis-0.1.0
annotations:
meta.helm.sh/release-name: tom-redis-custom-code
meta.helm.sh/release-namespace: default
managedFields:
- manager: Go-http-client
operation: Update
apiVersion: v1
time: '2020-08-28T22:49:27Z'
fieldsType: FieldsV1
fieldsV1:
'f:data':
.: {}
'f:redis-password': {}
'f:metadata':
'f:annotations':
.: {}
'f:meta.helm.sh/release-name': {}
'f:meta.helm.sh/release-namespace': {}
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/managed-by': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/version': {}
'f:helm.sh/chart': {}
'f:type': {}
data:
redis-password: dGhpc19pc19hX3Bhc3N3b3Jk
type: Opaque
In this case, when I manually create a (fake) slow command using redis-cli -h 172.18.0.14 -p 6379 -a this_is_a_password DEBUG SLEEP 1, it can report to elasticsearch and I can see the log. However, at the same time, there are a lot of 2020-08-28T23:57:41.427Z ERROR harvester/registry.go:98 Error running input: error receiving slowlog data: dial tcp 127.0.0.1:6379: connect: connection refused errors in Filebeat logs.
If I comment the # FIRST line and uncommend the # SECOND line, then everything does not work. No slow commands will be reported to elasticsearch and I cannot see it using kibana discovery. However, there is no more error in Filebeat log output!
I would appreciate for any suggestions. Thanks!
p.s. Is there a general methodology to debug such cases? e.g. enabling some logging or debugging?
Thanks!
, but let's see what logs can provide.