Kubernetes filebeat not sending logs to secured Elasticsearch

[pratikshatiwari]

Hello Team

We are facing an issue where we have setup the filebeat on kubernetes environment it is working well if we setup the output without https but while setting the output as secured elasticsearch node i.e. https://elastichost:9200 it doesnt go, followed below link to setup filebeat

(Run Filebeat on Kubernetes | Filebeat Reference [master] | Elastic)

follwoing are the configuration from filebeat-kubernetes.yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:XX.XX.XX.XX}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.certificate_authorities: ["/etc/ssl/certs/ca.crt"]
      ssl_verify: true


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.4.3
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: https://XX.XX.XX.XX
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: elastic
        - name: ELASTICSEARCH_SCHEME
          value: "https"
        - name: ELASTICSEARCH_SSL_VERIFY
          value: "true"

Regards
Pratiksha

[Ehsan_Orve]

Hi
I thing you should mount your public key in your pod as a local storage then provide a link to the key in filebet DaemonSet.

[pratikshatiwari]

Hi @Ehsan_Orve

Thank you for revert i am not able to get the configuration procedure as you advised,if possible could you please help with steps how i can achieve it in my kubernetes setup

kubernetes version 1.21

Also do you fidn any issue issue my ssl configuration from above configuration

Regards
Pratiksha

[Ehsan_Orve]

I am sorry i tought you have used elastic agent. It is better and also easier to use Fleet and elastic agent to gathering and monitoring k&s pods.

about ssl did you use elasticsearch-certutil to generate a ssl? If yes, you should address elasticseach-ca.pem file (If I can remember correctly) in filebeat config file.

can u access to elasticsearch on port 9200 securly in a web browser?

[pratikshatiwari]

Hello @Ehsan_Orve

i am using filebeat-kubernetes.yanl file where i did not get the option to add certificate information

step1
curl -L -O https://raw.githubusercontent.com/elastic/beats/8.4/deploy/kubernetes/filebeat-kubernetes.yaml

step2
after downloading file i editing the yaml file stored at path c:/code
NOTE: without SSL

step3
kubectl create -f filebeat-kubernetes.yaml

step4
kubectl apply -f filebeat-kubernetes.yaml

i am following above to setup filebeat on kubernetes cluster

Yes i generated the ssl certificate using elastic-cert utility
i do not get the right option in filebeat-kubernetes.yaml to configure ssl information

Yes i can access elasticsearch on port 9200 in web browser

Regards
Pratiksha

[Ehsan_Orve]

Hi @pratikshatiwari

I think u should use VolumeMount and VolumePath in Daemonset under spec-->containers section:
In this way you will attach crt file into the filebeat container then use mount it in "/etc/ssl/certs/ca.crt" so filebeat can use it. Maybe it seems a little difficult to put this lines in your DaemonSet file, but you can do it.

volumeMounts:
   - name: cert-file
     mountPath: /etc/ssl/certs/ca.crt 
     readOnly: true
volumes:
  - name: cert-file
     hostPath: 
             path: "Path of crt file on your kubernetes node"

[pratikshatiwari]

Hello @Ehsan_Orve

Thank you for sharing the configuration, however i have started getting the configuration error
Am i doing anything wrong here?

Following configuration i made
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:8.4.3
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
volumeMounts:
- name: cert-file
mountPath: /etc/ssl/certs/ca.crt
readOnly: true
volumes:
- name: cert-file
hostPath:
path: "/etc/ssl/certs/ca.crt"

Error detail as below:

serviceaccount/filebeat unchanged
The request is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"apps/v1","kind":"DaemonSet","metadata":{"annotations":{},"labels":{"k8s-app":"filebeat"},"name":"filebeat","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"filebeat"}},"template":{"metadata":{"labels":{"k8s-app":"filebeat"}},"spec":{"containers":[{"args":["-c","/etc/filebeat.yml","-e"],"env":[{"name":"ELASTICSEARCH_HOST","value":"https://35.223.228.215"},{"name":"ELASTICSEARCH_PORT","value":"9200"},{"name":"ELASTICSEARCH_USERNAME","value":"elastic"},{"name":"ELASTICSEARCH_PASSWORD","value":"elastic"},{"name":"ELASTICSEARCH_SCHEME","value":"https"},{"name":"ELASTICSEARCH_SSL_VERIFY","value":"true"},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"fieldPath":"spec.nodeName"}}}],"image":"docker.elastic.co/beats/filebeat:8.4.3","name":"filebeat","resources":{"limits":{"memory":"200Mi"},"requests":{"cpu":"100m","memory":"100Mi"}},"securityContext":{"runAsUser":0},"volumeMounts":[{"mountPath":"/etc/filebeat.yml","name":"config","readOnly":true,"subPath":"filebeat.yml"},{"mountPath":"/usr/share/filebeat/data","name":"data"},{"mountPath":"/var/lib/docker/containers","name":"varlibdockercontainers","readOnly":true},{"mountPath":"/var/log","name":"varlog","readOnly":true}],"volumes":[{"hostPath":{"path":"/etc/ssl/certs/ca.crt"},"name":"cert-file"}]}],"dnsPolicy":"ClusterFirstWithHostNet","hostNetwork":true,"serviceAccountName":"filebeat","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":416,"name":"filebeat-config"},"name":"config"},{"hostPath":{"path":"/var/lib/docker/containers"},"name":"varlibdockercontainers"},{"hostPath":{"path":"/var/log"},"name":"varlog"},{"hostPath":{"path":"/var/lib/filebeat-data","type":"DirectoryOrCreate"},"name":"data"}]}}}}\n]] spec:map[template:map[spec:map]]]": strict decoding error: unknown field "spec.template.spec.containers[0].volumes"

Regards
Pratiksha

[Ehsan_Orve]

Could u copy all daemonSet here?

[Ehsan_Orve]

And also read this:

Filebeat running on Kubernetes connected to Elasticsearch using SSL

I would like to use filebeat on Kubernetes and connect it to a secured Elasticsearch server, using certificates. I did a search to find how to do this but only found an un-answered related question. After some test I got it, so I will share my findings to help anybody else. These are the steps: Follow the original documentation to download the yaml file. As shown on the documentation, update this file with the correct details about the Elasticsearch server: hostname, port, user and passw…