Filebeat not sending logs to EKS in a Microk8s Kubernetes environment

NashF · October 17, 2023, 2:20pm

I'm at my wits end with this ELK stack and Filebeat. Currently using Elasticsearch, Kibana ,Filebeat all version 8.10.2 and logstash logstash-logback-encoder:7.4 for converting my springboot microservice logs into json objects

Currently the Elasticsearch, Kibana are working just fine with each other, I'm able to access the Kibana dashboard. No issue.

The real issue is Filebeat, for all intents and purposes it seems to be functioning well at least from the logs, I'm not seeing any errors. From this I was assuming that Filebeat was able to send log data to Elasticsearch to be indexed. But no log data was being sent nor any indexes being created for Filebeat in Elastic Search.

I'm adding the manifests and Filebeat yml configuration below, Does filebeat not work nicely with ELK in a microk8 kubernetes environment?

================== Filebeat.yml start ================================
filebeat.autodiscover:
providers:
- type: kubernetes
include_labels:
- "app"
node: kvm-u-node0
templates:
- condition:
contains:
kubernetes.container.labels.collect_logs_with_filebeat: "true"
config:
- type: container
paths:
- "/var/lib/docker/containers/${data.kubernetes.namespace}${data.kubernetes.pod.name}${data.kubernetes.pod.uid}/${data.kubernetes.container.name}//*.log"
json.keys_under_root: true
json.add_error_key: true
json.message_key: message
- condition:
contains:
kubernetes.container.labels.decode_log_event_to_json_object: "true"
config:
- type: container
paths:
- "/var/lib/docker/containers/${data.kubernetes.namespace}${data.kubernetes.pod.name}${data.kubernetes.pod.uid}/${data.kubernetes.container.name}//*.log"
processors:
- decode_json_fields:
fields: ["message"]
target: ""
overwrite_keys: true

filebeat.inputs:

- type: log

enabled: true

paths:

- "/var/lib/docker/containers/${data.kubernetes.namespace}${data.kubernetes.pod.name}${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/**/*.log"

output.elasticsearch:
enabled: true
hosts: ["elasticsearch-service-for-springboot:9200"]
username: "elastic"
password: "password_elastic"
index: "filebeat-%{+yyyy.MM.dd}"
pipeline: "filebeat-import-example-pipeline"

setup.kibana:

host: "10.152.183.79:5601"

username: "kibana_system"

password: "password_kibana"

setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "filebeat-{now/d}-000001"

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"

logging.level: debug
logging.metrics.enabled: false
================== Filebeat.yml end ================================
==================Filebeat manifest start ============================

apiVersion: v1
kind: Service
metadata:
name: filebeat-service
namespace: dev
spec:
selector:
app: filebeat
ports:
- protocol: TCP
port: 5044
targetPort: 5044
type: ClusterIP

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: dev
labels:
app: filebeat
spec:
selector:
matchLabels:
app: filebeat
template:
metadata:
labels:
app: filebeat
spec:
hostNetwork: true
volumes:
- name: config
hostPath:
path: /home/nash/Documents/food-ordering-demo-microservices/elasticsearch/kubernetes/logging/filebeat/filebeat.yml
type: File
- name: varlibdockercontainers
hostPath:
path: /var/log/pods
- name: kuberenetes-sock
hostPath:
path: /var/run/kubernetes.sock
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:8.10.2
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch-service-for-springboot
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: password_elastic
volumeMounts:
- name: config
mountPath: /usr/share/filebeat/filebeat.yml
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: kuberenetes-sock
mountPath: /var/run/kubernetes.sock
readOnly: true
==================Filebeat manifest end ============================

==================Elasticsearch and Kibana manifests start ====================

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: dev
spec:
serviceName: elasticsearch
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:

initContainers:

- name: init-reset-password

image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2

command: [ "/bin/sh", "-c", "cp /config/reset-passwords.sh /usr/share/elasticsearch/bin/ && chmod +x /usr/share/elasticsearch/bin/reset-passwords.sh && /usr/share/elasticsearch/bin/reset-passwords.sh" ]

  containers:
    - name: elasticsearch
      image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
      imagePullPolicy: Always

command: [ "/bin/sh", "-c", "cp /config/reset-passwords.sh /usr/share/elasticsearch/bin/ && chmod +x /usr/share/elasticsearch/bin/reset-passwords.sh && /usr/share/elasticsearch/bin/reset-passwords.sh" ]

      resources:
        requests:
          memory: "600Mi"
          cpu: ".5"
        limits:
          memory: "2Gi"
          cpu: "1.5"
      env:
        - name: cluster.routing.allocation.disk.threshold_enabled
          value: "false"
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.security.enrollment.enabled
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "false"
        - name: xpack.security.http.ssl.enabled
          value: "false"
        - name: discovery.type
          value: single-node
        - name: node.name
          value: elasticsearch
        - name: bootstrap.memory_lock
          value: "false"
        - name: ES_JAVA_OPTS
          value: -Xms1g -Xmx1g
        - name: ELASTICSEARCH_PASSWORD
          value: password_elastic
        - name: KIBANA_PASSWORD
          value: password_kibana
      ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
      volumeMounts:
        - name: data
          mountPath: /home/nash/elasticsearch/data

- name: init-config

mountPath: /config

volumes:

- name: init-config

configMap:

name: elasticsearch-init-config

  securityContext:
    runAsUser: 1000  # Set the desired user ID
    fsGroup: 1000    # Set the desired group ID

volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi

apiVersion: v1
kind: Service
metadata:
name: elasticsearch-service-for-springboot
labels:
app: elasticsearch-service-for-springboot
namespace: dev
spec:
ports:
- name: elasticsearch-port1
port: 9200
targetPort: 9200
- name: elasticsearch-port2
port: 9300
targetPort: 9300
type: ClusterIP
selector:
app: elasticsearch

apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: dev
spec:
selector:
matchLabels:
app: kibana
replicas: 1
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:8.10.2 # Specify the appropriate Kibana version
imagePullPolicy: Always
env:
- name: server.port
value: "5601"
- name: xpack.security.enabled
value: "false"
- name: xpack.ingestManager.fleet.tlsCheckDisabled
value: "false"
- name: xpack.fleet.enabled
value: "true"
- name: xpack.security.transport.ssl.enabled
value: "false"
- name: xpack.security.http.ssl.enabled
value: "false"
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch-service-for-springboot:9200 # URL of the Elasticsearch service
- name: xpack.fleet.registryUrl
value: https://epr.elastic.co/
- name: ELASTICSEARCH_USERNAME
value: kibana_system
- name: ELASTICSEARCH_PASSWORD
value: password_kibana
ports:
- containerPort: 5601

apiVersion: v1
kind: Service
metadata:
name: kibana-service-for-springboot
labels:
app: kibana-service-for-springboot
namespace: dev
spec:
ports:
- name: kibana-port1
port: 5601
targetPort: 5601
type: ClusterIP
selector:
app: kibana

==================Elasticsearch and Kibana manifests end ====================

Snippet of the podlogs in Filebeat :

system · November 14, 2023, 4:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat doesn't send logs to Elasticsearch Beats filebeat	8	2352	February 16, 2021
How to check logs were sending to elasticsearch Beats filebeat	12	1406	May 20, 2022
Filebeat is not sending logs to Elasticsearch Beats docker , filebeat	4	597	April 28, 2024
Trouble with sending logs from filebeat to elasticsearch Beats beats-module , filebeat	8	740	December 20, 2021
Filebeat not sending data to elasticsearch Beats filebeat	3	662	August 8, 2023

Filebeat not sending logs to EKS in a Microk8s Kubernetes environment

filebeat.inputs:

- type: log

enabled: true

paths:

- "/var/lib/docker/containers/${data.kubernetes.namespace}${data.kubernetes.pod.name}${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/**/*.log"

setup.kibana:

host: "10.152.183.79:5601"

username: "kibana_system"

password: "password_kibana"

logging.level: debug logging.metrics.enabled: false ================== Filebeat.yml end ================================ ==================Filebeat manifest start ============================

apiVersion: v1 kind: Service metadata: name: filebeat-service namespace: dev spec: selector: app: filebeat ports: - protocol: TCP port: 5044 targetPort: 5044 type: ClusterIP

==================Elasticsearch and Kibana manifests start ====================

initContainers:

- name: init-reset-password

image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2

command: [ "/bin/sh", "-c", "cp /config/reset-passwords.sh /usr/share/elasticsearch/bin/ && chmod +x /usr/share/elasticsearch/bin/reset-passwords.sh && /usr/share/elasticsearch/bin/reset-passwords.sh" ]

command: [ "/bin/sh", "-c", "cp /config/reset-passwords.sh /usr/share/elasticsearch/bin/ && chmod +x /usr/share/elasticsearch/bin/reset-passwords.sh && /usr/share/elasticsearch/bin/reset-passwords.sh" ]

- name: init-config

mountPath: /config

volumes:

- name: init-config

configMap:

name: elasticsearch-init-config

Related topics

logging.level: debug
logging.metrics.enabled: false
================== Filebeat.yml end ================================
==================Filebeat manifest start ============================

apiVersion: v1
kind: Service
metadata:
name: filebeat-service
namespace: dev
spec:
selector:
app: filebeat
ports:
- protocol: TCP
port: 5044
targetPort: 5044
type: ClusterIP