ECK - quickstart - heartbeat example disrupts filebeat example

[joachimnielandt]

Hello all,

I have a question that is more about me trying to understand ECK than it is an issue. In trying to run elastic+kibana+filebeat+heartbeat, I find that I cannot run both beats together.

I'm testing out a self-hosted ECK stack on Azure AKS. In following the tutorial, I created a namespace future-test in which I run the following templates:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.3.2
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.3.2
  count: 1
  elasticsearchRef:
    name: quickstart

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 8.3.2
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
  daemonSet:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

So far so good. I can port forward into Kibana and see the filebeat time series created and being updated. Pods are behaving well. Now, I want to integrate heartbeat as well:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: heartbeat
  version: 8.3.2
  elasticsearchRef:
    name: quickstart
  config:
    heartbeat.monitors:
    - type: tcp
      schedule: '@every 5s'
      hosts: ["quickstart-es-http.future-test.svc:9200"]
  deployment:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        securityContext:
          runAsUser: 0

Note I modified the default in the host example to my own namespace. After a couple of seconds, I can see the filebeat data stop coming in in Kibana, and heartbeat data appear. In the filebeat pod I see these errors appear:
{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [future-test-quickstart-beat-user] with roles [beats_admin,eck_beat_es_heartbeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [filebeat-8.3.2,.ds-filebeat-8.3.2-2022.07.19-000001], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

Is the deployment of heartbeat messing with the elasticsearch users, such that filebeat cannot auth anymore? Am I misusing the templates (use the same name erroneously)? Any help or example would be greatly appreciated, thanks!