Filebeat not shipping logs to Elasticsearch service on K8S

Elastic Stack Beats
,
1

Hello,
I have deployed Filebeat using the Helm chart on K8s as a daemonset. We would like to use Filebeat to send Kubelet, Containerd, and docker logs to Elasticsearch Service. The config yaml is below:

filebeat.inputs:  
            - type: journald
              id: service-kubelet
              include_matches.match:
                - _SYSTEMD_UNIT=kubelet.service
            - type: journald
              id: service-containerd
              include_matches.match:
                - _SYSTEMD_UNIT=containerd.service
            - type: journald
              id: service-docker
              include_matches.match:
                - _SYSTEMD_UNIT=docker.service 

          output.elasticsearch:
            hosts: '["https://${ELASTICSEARCH_HOSTS:address.us-east-1.aws.found.io:9243}"]'
            protocol: https
            username: '${ELASTICSEARCH_USERNAME}'
            password: '${ELASTICSEARCH_PASSWORD}'
            indices:
              - index: "k8s-node-logs-%{[agent.version]}-%{+yyyy.MM.dd}"
            setup.template.enabled: false

I don't see the index created in ELK when I login. The logs also does not provide any info on how to troubleshoot this. Running the following command while execing into the pods returns ok:
filebeat test config
filebeat test output
Can you please provide some help?

2

Hi @Max9998 Welcome to the community.

What version of the stack?

What do the filebeat logs show?

Take out these lines and try again... there are more settings if you want to change the index name (we can solve that later)

Come back with this info and let us know.

3

Hello - Thanks for the response
The version of stack we use is 8.2.3. The logs don't show anything helpful. If I remove the index option how do I know where the logs are going to?

Logs:

{"log.level":"info","@timestamp":"2023-01-02T15:10:39.918Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/ │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.919Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: b248637c-64b1-4784-995d-7225c85a7d36","service.name":"filebeat","ecs.version":"1.6.0"}      │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.921Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","service.name":"filebeat","ecs.version":"1.6.0"}             │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.versi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/ │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"f81376b │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","a │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.923Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86 │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: filebeat; Version: 8.5.1","service.name":"filebeat","ecs.version":"1.6.0"}               │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)","service.name" │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.929Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://url.us-e │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.929Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: filebeat-filebeat-pvmfj","service.name":"filebeat","ecs.versio │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0" │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6 │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}                            │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.167Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transa │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.versi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}           │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.id filebeat.i │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 1)","service.name":"filebeat","ecs.ver │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"fi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"test-fu │
│ {"log.level":"info","@timestamp":"2023-01-02T15:11:09.934Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metri │
│
4

Hi @Max9998

The logs will go to the data stream

filebeat-8.2.3 by default.

  1. I am not an expert on the journals input can you try just the base config.
filebeat.inputs:
- type: journald
  id: everything
  1. please share the entire filebeat config properly formatted not just the snippets we can not see the provider you are using ...
5

Hi @stephenb,
I tried just base config for journald - no dice. The full config for filebeat is below that excludes the index part you asked me to remove:

filebeat.yml: |
    filebeat.inputs:
      - type: journald
        id: service-kubelet
        include_matches.match:
          - _SYSTEMD_UNIT=kubelet.service
      - type: journald
        id: service-containerd
        include_matches.match:
          - _SYSTEMD_UNIT=containerd.service
      - type: journald
        id: service-docker
        include_matches.match:
          - _SYSTEMD_UNIT=docker.service 

    output.elasticsearch:
      hosts: '["https://${ELASTICSEARCH_HOSTS:address.us-east-1.aws.found.io:9243}"]'
      protocol: https
      username: '${ELASTICSEARCH_USERNAME}'
      password: '${ELASTICSEARCH_PASSWORD}'
6

Sorry I am not a helm expert... So that deploys a daemonset? Ie a filebeat pod on each node?

If you exec into the filebeat containers and run journalctl .. do you get output?

If you run this from inside the filebeat container...

journalctl -o json

Do you get output...

There is something basic missing...

Something makes me think that journalctl from inside the filebeat container will not read the host / node journalctl

You could also just try the container or syslog input to see if you are actually collecting logs and Shipping them.

Oh also set the login level to debug so we can see more messages.

logging.level: debug

7

filebeat is deployed using helm as a daemonset on Kubernetes cluster. I execed into one of the pods and I found out it doesn't have journalctl installed. I went ahead and created a custom image and installed journald but doesn't look like it is recording any logs.

You may be correct on this " Something makes me think that journalctl from inside the filebeat container will not read the host / node journalctl"

8

What does the DaemonSet manifest look like? Is the host path of the container mounted into the pods?

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.