# Pod container logs stop randomly

**URL:** <https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632>\
**Category:** Elastic Agent\
**Created:** [February 15, 2023, 2:54pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632 "2023-02-15T14:54:40Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![clewo](https://avatars.discourse-cdn.com/v4/letter/c/5fc32e/32.png) [@clewo](https://discuss.elastic.co/u/clewo)\
**Post date:** [February 16, 2023, 10:51am UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/2 "2023-02-16T10:51:59Z")

</div>

This sounds similar but not identically to the topics blow. Might be related somehow:

> <https://github.com/elastic/elastic-agent/issues/2269>
>
> Since upgrading to Elastic Agent v8.6.x, I've started using condition-based auto…discovery for Kubernetes pods in Elastic Agent. When pod logs rotate, they use a \`copytruncate\` strategy. Elastic Agent continues ingesting the logs as expected when this happens. However, when a new pod starts, Elastic Agent does not appear to start monitoring until the agent is restarted. This includes when:
> 
> \* A new deployment happens on a node for the first time (i.e., no prior instances of that deployment were running).
> \* A pod restarts on the same node.
> \* A pod stops on one node and a new instance starts on another node (i.e., when a pod "switches nodes").
> 
> This has been observed with Elastic Agent v8.6.0 and v8.6.1 in EKS on Kubernetes 1.22.14.
> 
> I have also tested this with \`filebeat-8.6.1\` deployed to my cluster using the older-style \`filebeat.autodiscover.providers\` hints-based autodiscovery, and it \*\*does not\*\* appear to exhibit the same behavior.
> 
> I reported this issue in the \[forums\]\[1\] and received a response from another user who had \[reported the same issue\]\[2\] in the forums about the same time I did.
> 
> Per the other user's recommendation, I downgraded Elastic Agent to v8.5.3, which appears to have resolved the issue.
> 
> To reproduce:
> 
> \* Deploy Elastic Agent Standalone 8.6.0 or 8.6.1 with a condition-based filestream input.
> \* Create a new deployment with a container that matches the filestream input configuration \*\*after\*\* Elastic Agent is running and logging has started.
> 
> Here is a sample input configuration, taken from a pod for which this is particularly noticeable, since it generally terminates after about two hours and a new instance is scheduled to another node:
> 
> \`\`\`yaml
> \---
> inputs:
> - id: 'filestream-myapp-0d9ea285-223f-4b6a-9207-9432b0eac168'
> name: 'filestream-myapp'
> type: 'filestream'
> use\_output: 'default'
> data\_stream:
> namespace: 'default'
> streams:
> # mycontainer
> - id: 'logs-${kubernetes.pod.name}-${kubernetes.container.id}'
> condition: '${kubernetes.container.name} == "mycontainer"'
> data\_stream:
> dataset: 'myapp.log'
> type: 'logs'
> parsers:
> - container:
> stream: 'all'
> format: 'auto'
> - ndjson:
> expand\_keys: true
> ignore\_decoding\_error: true
> overwrite\_keys: true
> target: ''
> paths: \['/var/log/containers/\*${kubernetes.container.id}.log'\]
> pipeline: 'logs-myapp.log'
> processors:
> - add\_locale:
> format: 'offset'
> prospector.scanner.symlinks: true
> tags: \['myapp'\]
> \`\`\`
> 
> I do \*\*not\*\* currently have \`providers.kubernetes.hints.enabled: true\` set (the documentation for conditions-based autodiscovery does not require it), but I have tested it both ways with the same result.
> 
> \[1\]:https://discuss.elastic.co/t/elastic-agent-conditions-based-autodiscover-doesnt-pick-up-newly-scheduled-pods-containers
> \[2\]:https://discuss.elastic.co/t/elastic-agent-autodiscovery-for-kubernetes-pod-logs-not-working-after-upgrading-to-8-6-x

> [@Elastic Agent conditions-based autodiscover doesn't pick up newly-scheduled pods/containers](https://discuss.elastic.co/t/elastic-agent-conditions-based-autodiscover-doesnt-pick-up-newly-scheduled-pods-containers/325271/4):
>
> After downgrading to 8.5.3, I let it run overnight, and everything is ingesting as inspected. It doesn't look as if any bugs are open on this, so I've submitted a bug report: [Elastic Agent 8.6.x standalone deployment in Kubernetes doesn't start monitoring new pods until agent restart · Issue #2269 · elastic/elastic-agent · GitHub](https://github.com/elastic/elastic-agent/issues/2269)

> [@Elastic-Agent autodiscovery for Kubernetes pod logs not working after upgrading to 8.6.x](https://discuss.elastic.co/t/elastic-agent-autodiscovery-for-kubernetes-pod-logs-not-working-after-upgrading-to-8-6-x/325227/2):
>
> For anybody reading this post having the same issue, it's going to be tracked here: [Elastic Agent 8.6.x standalone deployment in Kubernetes doesn't start monitoring new pods until agent restart · Issue #2269 · elastic/elastic-agent · GitHub](https://github.com/elastic/elastic-agent/issues/2269)

Does restarting the elastic-agents solve the issue? Does downgrading the agents to 8.5.3 solve the issue?

---

_[View the full topic](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632)._