Kubernetes deployed Elastic Agent is looped in beat restart

Alphayeeeet · January 10, 2024, 8:11am

Hello,

I tried to deploy Fleet-managed Elastic Agent with Kubernetes Integration into RedHat OpenShift. When deploying an agent policy update using Fleet, the beats are caught up in a restart loop:

Agent version: beats/elastic-agent:8.11.2

After restarting the agent pods manually , the problem is solved.

This might be a bug in Elastic Agent.

I would appreciate any advice. Thank you.

Alphayeeeet · January 15, 2024, 7:50am

Reposted: Kubernetes deployed Elastic Agent is looped in beat restart

stephenb · January 15, 2024, 5:15pm

Hi @Alphayeeeet

Welcome to the community.

This is often caused by an out-of-memory issue. You might want to bump up the resources for the agent.

Also, I am going to close the other Topic as this is a fine / close enough category.

Alphayeeeet · January 22, 2024, 12:39pm

Hi @stephenb

Thank you for the reply. After increasing the memory limits, the agent is running smoothly. Maybe this could be added in the troubleshooting documentation of Elastic Agent?

stephenb · January 23, 2024, 7:58am

Actually there is a open issue to make this much easier to identify.

github.com/elastic/elastic-agent

Report the agent status as degraded when previously OOMKilled on Kubernetes

opened 08:30PM - 19 Oct 23 UTC

cmacknz

Team:Elastic-Agent Team:Cloudnative-Monitoring

We need to make it easier to detect inadequate memory limits on Kubernetes, whic…h are extremely common. The agent should detect when its last status was OOM killed and report its status as degraded. Detecting that an agent has been OOMKilled from diagnostics along is not easy, it must be inferred from process restarts appearing the agent diagnostics with no other plausible explanations. Today the primary way for us to detect this is to instruct users to run `kubectl describe pod` and look for the following: ``` Last State: Terminated Reason: OOMKilled Exit Code: 137 ``` We should automate this process and have the agent read the last state and reason for itself and report it in the agent status report.

system · February 20, 2024, 7:58am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.