# Filebeat Not Shipping Container logs from Kubernetes **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [April 5, 2024, 8:40pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891 "2024-04-05T20:40:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 5, 2024, 8:40pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/1 "2024-04-05T20:40:11Z")
I tried the config settings for autodiscover described here: > **[Monitoring Kubernetes the Elastic way using Filebeat and Metricbeat](https://www.elastic.co/blog/monitoring-kubernetes-the-elastic-way-using-filebeat-and-metricbeat)** > > If you’re already using open source monitoring tools in your organization, you can use those tools alongside the Elastic Stack to monitor Kubernetes. But, if you’re new to Kubernetes monitoring, there... I confirmed that the pod whose log I want to capture does produce output when I run the command `kubectl logs [podName]` I also confirmed the existence of the logs (flat files) by executing kubectl exec. However, the logs do not appear to be shipped to Elasticsearch. When I try to create an index pattern, nothing matches in Kibana Stack Management nor does any results appear in Kibana Discover under the filebeat\* index pattern. Here are the relevant sections of the filebeat-kubernetes.yaml manifest file ```auto data: filebeat.yml: |- setup: template: name: "ams-rancher" pattern: "ams-rancher-%{+yyyyMMdd}" # filebeat.inputs: # - type: filestream # paths: # - /var/log/containers/*.log filebeat.autodiscover: providers: - type: kubernetes node: ${NODE_NAME} hints.enabled: true hints.default_config: type: filestream id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id} paths: - /opt/amsdev/logs/ams-cache-manager-logs/*.log parsers: - container: ~ prospector: scanner: fingerprint.enabled: true symlinks: true file_identity.fingerprint: ~ spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: filebeat image: docker.elastic.co/beats/filebeat:8.12.2 args: [ "-c", "/etc/filebeat.yml", "-e", ] env: - name: ELASTICSEARCH_HOST value: "{hostname}" - name: ELASTICSEARCH_PORT value: "9200" - name: ELASTICSEARCH_USERNAME value: - name: ELASTICSEARCH_PASSWORD value: - name: ELASTIC_CLOUD_ID value: - name: ELASTIC_CLOUD_AUTH value: - name: NODE_NAME value: vmdev-rms4 ``` ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 5, 2024, 9:55pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/2 "2024-04-05T21:55:31Z")
Related question: We actually have 6 nodes running in our Rancher/K8 setup. Would I enter all 6 node names into the NODE\_NAME config in the manifest file if we needed to monitor all of them? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 5, 2024, 10:11pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/3 "2024-04-05T22:11:09Z")
Hi @paolovalladolid That article is aging a bit, conceptually still good but I would follow this > **[Run Filebeat on Kubernetes | Filebeat Reference \[8.13\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html)** and use the suggested manifest... `https://raw.githubusercontent.com/elastic/beats/8.13/deploy/kubernetes/filebeat-kubernetes.yaml` Leave this as is.. that gets substituted at runtime. ` node: ${NODE_NAME}` ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 8, 2024, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/4 "2024-04-08T14:22:13Z")
Thank you! I renamed the manifest file I was using, and replaced it with a clean copy of the manifest file from the provided link. The only change I made to the clean copy was to the value of ELASTICSEARCH\_HOST. So my first goal was achieved which was to ship the container log to our ES cluster. Next is to configure the index pattern that we need to use. Will report back. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 8, 2024, 4:07pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/5 "2024-04-08T16:07:38Z")
> [@paolovalladolid](#): > > Next is to configure the index pattern that we need to use. Will report back. You should look at this cool solution I just showed another .. .Or you will need to setup up all your custom templates, naming etc..etc.. which is fine but it is a lot of work. > [@Filebeat with multiple path and index](https://discuss.elastic.co/t/filebeat-with-multiple-path-and-index/356487/7): > > BTW you can also do this sending direct from Filebeat to Elasticsearch and you will still get all the benefits above that I mentioned. filebeat.inputs: - type: filestream id: my-filestream-id enabled: true paths: - /var/log/\*.log fields\_under\_root: true fields: data\_stream.type: logs data\_stream.dataset: system\_log data\_stream.namespace: default setup.ilm.enabled: false setup.template.enabled: false # setup.template.settings: # index.number\_of\_shards: 1 setup.kiban… ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 8, 2024, 7:10pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/6 "2024-04-08T19:10:30Z")
Thanks, Stephen! In our pre-Kubernetes/Rancher days, we had been collecting logs from 6 different inputs, and so we had 6 input entries in our old filebeat.yml file. What you advise for multiple input configuration looks similar enough, yet cleaner and more maintainable. I'll look into implementing this in our filebeat-kubernetes.yaml. ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 8, 2024, 9:55pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/7 "2024-04-08T21:55:56Z")
Whatever I'm trying to do to change the index pattern is not working. First I tried this config wth modified lines in bold: ```auto data: filebeat.yml: |- filebeat.inputs: - type: filestream id: kubernetes-container-logs paths: - /var/log/containers/*.log # NEW LINES BELOW fields_under_root: true fields: data_stream.type: logs data_stream.dataset: ams data_stream.namespace: rancher # NEW LINES ABOVE parsers: - container: ~ prospector: scanner: fingerprint.enabled: true symlinks: true file_identity.fingerprint: ~ processors: - add_kubernetes_metadata: host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" ``` and the ES output section ```auto output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}" ``` This resulted in no logs collected, with no index pattern matching "logs-\*" in Kibana. I then tried hardcoding the index property under output.elasticsearch but that did not work either. I then commented out the index line and was once again able to see logs in Kibana under the "filebeat-\*" index pattern. Do I need to put the setup.template lines back into my filebeat-kubernetes.yaml? That's the only thing I can think of that is breaking log collection from Kubernetes. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 9, 2024, 4:08am UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/8 "2024-04-09T04:08:36Z")
> [@paolovalladolid](#): > > Do I need to put the setup.template lines back into my filebeat-kubernetes.yaml? That's the only thing I can think of that is breaking log collection from Kubernetes. Yes I think you need, otherwise it will fail. ```auto setup.ilm.enabled: false setup.template.enabled: false ``` ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 9, 2024, 10:08pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/10 "2024-04-09T22:08:43Z")
I tried modifying the processors section to try to get only logs matching a pod name. This killed the log collection. ```auto processors: - add_kubernetes_metadata: host: ${NODE_NAME} indexers: - container_name: "ams-cache-manager-*" matchers: - logs_path: logs_path: "/var/log/containers/" ``` I would like to configure a separate Filebeat input per container so that we can have a separate dashboard in Kibana for each container. What is the correct way to set up one input per container? I also tried adding a line to the fields section instead of the processors section, like this - but log collection still did not resume. Do we need to switch to the autodiscover configuration? All examples of kubernetes.container.name being used in a manifest file only show this property under autodiscover. ```auto data: filebeat.yml: |- filebeat.inputs: - type: filestream id: ams-cache-manager-container-logs paths: - /var/log/containers/*.log fields_under_root: true fields: data_stream.type: logs data_stream.dataset: ams data_stream.namespace: rancher kubernetes.container.name: "ams-cache-manager-*" parsers: - container: ~ prospector: scanner: fingerprint.enabled: true symlinks: true file_identity.fingerprint: ~ processors: - add_kubernetes_metadata: host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 10, 2024, 2:34pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/11 "2024-04-10T14:34:37Z")
Hi @paolovalladolid Since this topic originally was solved I would recommend opening a new Topic with a New Subject and the details of your new issue. ---
**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Post date:** [April 10, 2024, 2:39pm UTC](https://discuss.elastic.co/t/filebeat-not-shipping-container-logs-from-kubernetes/356891/12 "2024-04-10T14:39:18Z")
Sorry about that. I literally just now figured out how to get log collection going again. This turned out to be the way - do the matching for the desired log source in the paths not the fields: ```auto filebeat.yml: |- filebeat.inputs: - type: filestream id: ams-cache-manager-container-logs paths: - /var/log/containers/ams-cache*.log fields_under_root: true fields: data_stream.type: logs data_stream.dataset: ams data_stream.namespace: rancher ``` Thanks for all the help!