# Pod container logs stop randomly

**URL:** <https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632>\
**Category:** Elastic Agent\
**Created:** [February 15, 2023, 2:54pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632 "2023-02-15T14:54:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![woodywoodsta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woodywoodsta/32/107962_2.png) [@woodywoodsta](https://discuss.elastic.co/u/woodywoodsta)\
**Post date:** [February 15, 2023, 2:54pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/1 "2023-02-15T14:54:41Z")

</div>

Since upgrading to 8.6.1 (from 8.5.x), I'm finding that logs that are supposed to be collected via the Kubernetes Integration for an elastic agent in Fleet mode are stopping completely and randomly (as far as I can see).

The logs stop for different pods at different times, and when they do stop, they start again after a few days, only to stop again after a day or two. Attached is a screenshot of logs from a particular pod.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/56255b1fa20d766cd2d9bbd25924a67f8eb4ab8b.png)

These events don't seem to correlate with any logs throughout the system, nor do they appear to occur at any time when changes are made, or upgrades. They do not correlate with pods deleting and creating nor when logs are rotated, so it's a real mystery to me at this point.

I should also mention that I am observing similar behaviour with a Custom Logs integration which I [originally thought had to do with leader election](https://github.com/elastic/integrations/issues/4466) - and so I'm wondering if this has something to do with `filebeat`.

Any help in starting to get to the bottom of this is greatly appreciated, otherwise using the Elastic Stack is sortof useless 😃

---

<div class="post-metadata">

**Author:** ![clewo](https://avatars.discourse-cdn.com/v4/letter/c/5fc32e/32.png) [@clewo](https://discuss.elastic.co/u/clewo)\
**Post date:** [February 16, 2023, 10:51am UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/2 "2023-02-16T10:51:59Z")

</div>

This sounds similar but not identically to the topics blow. Might be related somehow:

> <https://github.com/elastic/elastic-agent/issues/2269>
>
> Since upgrading to Elastic Agent v8.6.x, I've started using condition-based auto…discovery for Kubernetes pods in Elastic Agent. When pod logs rotate, they use a \`copytruncate\` strategy. Elastic Agent continues ingesting the logs as expected when this happens. However, when a new pod starts, Elastic Agent does not appear to start monitoring until the agent is restarted. This includes when:
> 
> \* A new deployment happens on a node for the first time (i.e., no prior instances of that deployment were running).
> \* A pod restarts on the same node.
> \* A pod stops on one node and a new instance starts on another node (i.e., when a pod "switches nodes").
> 
> This has been observed with Elastic Agent v8.6.0 and v8.6.1 in EKS on Kubernetes 1.22.14.
> 
> I have also tested this with \`filebeat-8.6.1\` deployed to my cluster using the older-style \`filebeat.autodiscover.providers\` hints-based autodiscovery, and it \*\*does not\*\* appear to exhibit the same behavior.
> 
> I reported this issue in the \[forums\]\[1\] and received a response from another user who had \[reported the same issue\]\[2\] in the forums about the same time I did.
> 
> Per the other user's recommendation, I downgraded Elastic Agent to v8.5.3, which appears to have resolved the issue.
> 
> To reproduce:
> 
> \* Deploy Elastic Agent Standalone 8.6.0 or 8.6.1 with a condition-based filestream input.
> \* Create a new deployment with a container that matches the filestream input configuration \*\*after\*\* Elastic Agent is running and logging has started.
> 
> Here is a sample input configuration, taken from a pod for which this is particularly noticeable, since it generally terminates after about two hours and a new instance is scheduled to another node:
> 
> \`\`\`yaml
> \---
> inputs:
> - id: 'filestream-myapp-0d9ea285-223f-4b6a-9207-9432b0eac168'
> name: 'filestream-myapp'
> type: 'filestream'
> use\_output: 'default'
> data\_stream:
> namespace: 'default'
> streams:
> # mycontainer
> - id: 'logs-${kubernetes.pod.name}-${kubernetes.container.id}'
> condition: '${kubernetes.container.name} == "mycontainer"'
> data\_stream:
> dataset: 'myapp.log'
> type: 'logs'
> parsers:
> - container:
> stream: 'all'
> format: 'auto'
> - ndjson:
> expand\_keys: true
> ignore\_decoding\_error: true
> overwrite\_keys: true
> target: ''
> paths: \['/var/log/containers/\*${kubernetes.container.id}.log'\]
> pipeline: 'logs-myapp.log'
> processors:
> - add\_locale:
> format: 'offset'
> prospector.scanner.symlinks: true
> tags: \['myapp'\]
> \`\`\`
> 
> I do \*\*not\*\* currently have \`providers.kubernetes.hints.enabled: true\` set (the documentation for conditions-based autodiscovery does not require it), but I have tested it both ways with the same result.
> 
> \[1\]:https://discuss.elastic.co/t/elastic-agent-conditions-based-autodiscover-doesnt-pick-up-newly-scheduled-pods-containers
> \[2\]:https://discuss.elastic.co/t/elastic-agent-autodiscovery-for-kubernetes-pod-logs-not-working-after-upgrading-to-8-6-x

> [@Elastic Agent conditions-based autodiscover doesn't pick up newly-scheduled pods/containers](https://discuss.elastic.co/t/elastic-agent-conditions-based-autodiscover-doesnt-pick-up-newly-scheduled-pods-containers/325271/4):
>
> After downgrading to 8.5.3, I let it run overnight, and everything is ingesting as inspected. It doesn't look as if any bugs are open on this, so I've submitted a bug report: [Elastic Agent 8.6.x standalone deployment in Kubernetes doesn't start monitoring new pods until agent restart · Issue #2269 · elastic/elastic-agent · GitHub](https://github.com/elastic/elastic-agent/issues/2269)

> [@Elastic-Agent autodiscovery for Kubernetes pod logs not working after upgrading to 8.6.x](https://discuss.elastic.co/t/elastic-agent-autodiscovery-for-kubernetes-pod-logs-not-working-after-upgrading-to-8-6-x/325227/2):
>
> For anybody reading this post having the same issue, it's going to be tracked here: [Elastic Agent 8.6.x standalone deployment in Kubernetes doesn't start monitoring new pods until agent restart · Issue #2269 · elastic/elastic-agent · GitHub](https://github.com/elastic/elastic-agent/issues/2269)

Does restarting the elastic-agents solve the issue? Does downgrading the agents to 8.5.3 solve the issue?

---

<div class="post-metadata">

**Author:** ![woodywoodsta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woodywoodsta/32/107962_2.png) [@woodywoodsta](https://discuss.elastic.co/u/woodywoodsta)\
**Post date:** [February 23, 2023, 12:13pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/3 "2023-02-23T12:13:36Z")

</div>

Interesting, thank you for linking those.

I'll do some tests again, but I've not really managed to find a pattern as to when and why the logs are stopping. Some of them don't appear to be related to when pods recreate or restart. It's odd.

I see there is another related issue: [K8s Integration does not report correct container.id when container restarts · Issue #5348 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/5348)

Both issues are v8.6, so I'm wondering if there is a bunch of regressions all related to the same thing that have been introduced, and we're all experiencing subtly different results.

---

<div class="post-metadata">

**Author:** ![woodywoodsta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woodywoodsta/32/107962_2.png) [@woodywoodsta](https://discuss.elastic.co/u/woodywoodsta)\
**Post date:** [February 23, 2023, 12:14pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/4 "2023-02-23T12:14:36Z")

</div>

To add: one thing is for certain: I've been unable to find any output from the agents, beats or other elastic components which indicate what is going wrong - so it must be a silent failure somewhere.

---

<div class="post-metadata">

**Author:** ![Savva\_Morozov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/savva_morozov/32/116837_2.png) [@Savva\_Morozov](https://discuss.elastic.co/u/Savva_Morozov)\
**Post date:** [March 21, 2023, 9:10pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/5 "2023-03-21T21:10:04Z")

</div>

Hello! I think we are experiencing the same problem. Elastic Agent 8.6.2 in Fleet mode using Kubernetes integration is not collecting some logs from pods or sometimes not collecting logs from specific pods at all, even after changes in configuration and agent restarts. The logs of Elastic agents didn't show anything special that could explain this strange behaviour. Standard Filebeat seems to collect all the logs with no problems.

---

<div class="post-metadata">

**Author:** ![woodywoodsta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woodywoodsta/32/107962_2.png) [@woodywoodsta](https://discuss.elastic.co/u/woodywoodsta)\
**Post date:** [April 10, 2023, 5:20pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/6 "2023-04-10T17:20:23Z")

</div>

This appears to be resolved in v8.7.0.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2023, 5:20pm UTC](https://discuss.elastic.co/t/pod-container-logs-stop-randomly/325632/7 "2023-05-08T17:20:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.