# Filebeat not shipping logs to Elasticsearch service on K8S
**URL:** https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [December 30, 2022, 4:58pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227 "2022-12-30T16:58:44Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Max9998](https://discuss.elastic.co/u/Max9998)
#### Post date: [December 30, 2022, 4:58pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/1 "2022-12-30T16:58:44Z")
Hello,
I have deployed Filebeat using the Helm chart on K8s as a daemonset. We would like to use Filebeat to send Kubelet, Containerd, and docker logs to Elasticsearch Service. The config yaml is below:
```auto
filebeat.inputs:
- type: journald
id: service-kubelet
include_matches.match:
- _SYSTEMD_UNIT=kubelet.service
- type: journald
id: service-containerd
include_matches.match:
- _SYSTEMD_UNIT=containerd.service
- type: journald
id: service-docker
include_matches.match:
- _SYSTEMD_UNIT=docker.service
output.elasticsearch:
hosts: '["https://${ELASTICSEARCH_HOSTS:address.us-east-1.aws.found.io:9243}"]'
protocol: https
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
indices:
- index: "k8s-node-logs-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.enabled: false
```
I don't see the index created in ELK when I login. The logs also does not provide any info on how to troubleshoot this. Running the following command while execing into the pods returns ok:
filebeat test config
filebeat test output
Can you please provide some help?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [December 31, 2022, 10:44pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/2 "2022-12-31T22:44:07Z")
Hi @Max9998 Welcome to the community.
What version of the stack?
What do the filebeat logs show?
Take out these lines and try again... there are more settings if you want to change the index name (we can solve that later)
> [@Max9998](#):
>
> ```auto
> indices:
> - index: "k8s-node-logs-%{[agent.version]}-%{+yyyy.MM.dd}"
>
> ```
Come back with this info and let us know.
---
### Author:  [@Max9998](https://discuss.elastic.co/u/Max9998)
#### Post date: [January 2, 2023, 3:57pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/3 "2023-01-02T15:57:09Z")
> [@Max9998](#):
>
> ```auto
> - type: journald
> id: service-kubelet
> include_matches.match:
> - _SYSTEMD_UNIT=kubelet.service
> - type: journald
> id: service-containerd
> include_matches.match:
> - _SYSTEMD_UNIT=containerd.service
> - type: journald
> id: service-docker
> include_matches.match:
> - _SYSTEMD_UNIT=docker.service
>
> ```
Hello - Thanks for the response
The version of stack we use is 8.2.3. The logs don't show anything helpful. If I remove the index option how do I know where the logs are going to?
Logs:
```auto
{"log.level":"info","@timestamp":"2023-01-02T15:10:39.918Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/ │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.919Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: b248637c-64b1-4784-995d-7225c85a7d36","service.name":"filebeat","ecs.version":"1.6.0"} │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.921Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","service.name":"filebeat","ecs.version":"1.6.0"} │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.versi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/ │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"f81376b │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.922Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","a │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.923Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86 │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: filebeat; Version: 8.5.1","service.name":"filebeat","ecs.version":"1.6.0"} │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.924Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)","service.name" │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.929Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://url.us-e │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.929Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: filebeat-filebeat-pvmfj","service.name":"filebeat","ecs.versio │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0" │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6 │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:39.930Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"} │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.167Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transa │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.versi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"} │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.169Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.id filebeat.i │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 1)","service.name":"filebeat","ecs.ver │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"fi │
│ {"log.level":"info","@timestamp":"2023-01-02T15:10:40.170Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"test-fu │
│ {"log.level":"info","@timestamp":"2023-01-02T15:11:09.934Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metri │
│
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 2, 2023, 4:15pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/4 "2023-01-02T16:15:38Z")
Hi @Max9998
1.
> [@Max9998](#):
>
> If I remove the index option how do I know where the logs are going to?
The logs will go to the data stream
`filebeat-8.2.3` by default.
1. I am not an expert on the journals input can you try just the base config.
```auto
filebeat.inputs:
- type: journald
id: everything
```
1. please share the entire filebeat config properly formatted not just the snippets we can not see the provider you are using ...
---
### Author:  [@Max9998](https://discuss.elastic.co/u/Max9998)
#### Post date: [January 3, 2023, 1:09am UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/5 "2023-01-03T01:09:28Z")
Hi @stephenb,
I tried just base config for journald - no dice. The full config for filebeat is below that excludes the index part you asked me to remove:
```auto
filebeat.yml: |
filebeat.inputs:
- type: journald
id: service-kubelet
include_matches.match:
- _SYSTEMD_UNIT=kubelet.service
- type: journald
id: service-containerd
include_matches.match:
- _SYSTEMD_UNIT=containerd.service
- type: journald
id: service-docker
include_matches.match:
- _SYSTEMD_UNIT=docker.service
output.elasticsearch:
hosts: '["https://${ELASTICSEARCH_HOSTS:address.us-east-1.aws.found.io:9243}"]'
protocol: https
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 3, 2023, 1:49am UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/6 "2023-01-03T01:49:40Z")
Sorry I am not a helm expert... So that deploys a daemonset? Ie a filebeat pod on each node?
If you exec into the filebeat containers and run `journalctl` .. do you get output?
If you run this from inside the filebeat container...
`journalctl -o json`
Do you get output...
There is something basic missing...
Something makes me think that `journalctl` from inside the filebeat container will not read the host / node `journalctl`
You could also just try the `container` or `syslog` input to see if you are actually collecting logs and Shipping them.
Oh also set the login level to debug so we can see more messages.
`logging.level: debug`
---
### Author:  [@Max9998](https://discuss.elastic.co/u/Max9998)
#### Post date: [January 3, 2023, 7:32pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/7 "2023-01-03T19:32:47Z")
> [@Max9998](#):
>
> ```auto
> - type: journald
> id: service-kubelet
> include_matches.match:
> - _SYSTEMD_UNIT=kubelet.service
> - type: journald
> id: service-containerd
> include_matches.match:
> - _SYSTEMD_UNIT=containerd.service
> - type: journald
> id: service-docker
> include_matches.match:
> - _SYSTEMD_UNIT=docker.service
>
> ```
filebeat is deployed using helm as a daemonset on Kubernetes cluster. I execed into one of the pods and I found out it doesn't have journalctl installed. I went ahead and created a custom image and installed journald but doesn't look like it is recording any logs.
You may be correct on this " Something makes me think that `journalctl` from inside the filebeat container will not read the host / node `journalctl`"
---
### Author:  [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [January 5, 2023, 3:28am UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/8 "2023-01-05T03:28:23Z")
What does the DaemonSet manifest look like? Is the host path of the container mounted into the pods?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 2, 2023, 5:28am UTC](https://discuss.elastic.co/t/filebeat-not-shipping-logs-to-elasticsearch-service-on-k8s/322227/9 "2023-02-02T05:28:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.