clewo
February 16, 2023, 10:51am
2
This sounds similar but not identically to the topics blow. Might be related somehow:
opened 01:53PM - 14 Feb 23 UTC
closed 09:15PM - 10 Mar 23 UTC
bug
Team:Elastic-Agent
Team:Cloudnative-Monitoring
8.7-candidate
v8.6.0
Since upgrading to Elastic Agent v8.6.x, I've started using condition-based auto… discovery for Kubernetes pods in Elastic Agent. When pod logs rotate, they use a `copytruncate` strategy. Elastic Agent continues ingesting the logs as expected when this happens. However, when a new pod starts, Elastic Agent does not appear to start monitoring until the agent is restarted. This includes when:
* A new deployment happens on a node for the first time (i.e., no prior instances of that deployment were running).
* A pod restarts on the same node.
* A pod stops on one node and a new instance starts on another node (i.e., when a pod "switches nodes").
This has been observed with Elastic Agent v8.6.0 and v8.6.1 in EKS on Kubernetes 1.22.14.
I have also tested this with `filebeat-8.6.1` deployed to my cluster using the older-style `filebeat.autodiscover.providers` hints-based autodiscovery, and it **does not** appear to exhibit the same behavior.
I reported this issue in the [forums][1] and received a response from another user who had [reported the same issue][2] in the forums about the same time I did.
Per the other user's recommendation, I downgraded Elastic Agent to v8.5.3, which appears to have resolved the issue.
To reproduce:
* Deploy Elastic Agent Standalone 8.6.0 or 8.6.1 with a condition-based filestream input.
* Create a new deployment with a container that matches the filestream input configuration **after** Elastic Agent is running and logging has started.
Here is a sample input configuration, taken from a pod for which this is particularly noticeable, since it generally terminates after about two hours and a new instance is scheduled to another node:
```yaml
---
inputs:
- id: 'filestream-myapp-0d9ea285-223f-4b6a-9207-9432b0eac168'
name: 'filestream-myapp'
type: 'filestream'
use_output: 'default'
data_stream:
namespace: 'default'
streams:
# mycontainer
- id: 'logs-${kubernetes.pod.name}-${kubernetes.container.id}'
condition: '${kubernetes.container.name} == "mycontainer"'
data_stream:
dataset: 'myapp.log'
type: 'logs'
parsers:
- container:
stream: 'all'
format: 'auto'
- ndjson:
expand_keys: true
ignore_decoding_error: true
overwrite_keys: true
target: ''
paths: ['/var/log/containers/*${kubernetes.container.id}.log']
pipeline: 'logs-myapp.log'
processors:
- add_locale:
format: 'offset'
prospector.scanner.symlinks: true
tags: ['myapp']
```
I do **not** currently have `providers.kubernetes.hints.enabled: true` set (the documentation for conditions-based autodiscovery does not require it), but I have tested it both ways with the same result.
[1]:https://discuss.elastic.co/t/elastic-agent-conditions-based-autodiscover-doesnt-pick-up-newly-scheduled-pods-containers
[2]:https://discuss.elastic.co/t/elastic-agent-autodiscovery-for-kubernetes-pod-logs-not-working-after-upgrading-to-8-6-x
Does restarting the elastic-agents solve the issue? Does downgrading the agents to 8.5.3 solve the issue?
2 Likes