Filebeat-json.log size and rotation

[greggailly (Grégoire Gailly)]

We are having size issues with the filebeat-json.log file.
One of our servers had one 122gb filebeat-json.log, are they not supposed to rotate to prevent this ?
All of our servers run windows server 2016/2019 and elastic agents are at version 7.14.1.

More interestingly all servers don't seem to behave the same way.
Some will rotate the files but we can't seem to figure out according to which criteria while other don't rotate at all.

Example 1:

Example 2:

Thanks for the help !

[ruflin]

The files should rotate. Do you use Elastic Agent standalone or managed mode? Could you share how you start Elastic Agent?

That the log files are not rotated might indicate a permission issue. Are all the Elastic Agents started with the same user on the system? Are the log files in the same directories?

Could you share your elastic-agent log file?

[greggailly (Grégoire Gailly)]

Agent is started in managed mode with the standard install command (with a manual restart due to enrollment issues as mentioned here: Elastic Agent won't enroll - #12 by greggailly).
All agents are started on different servers with different users so user permissions might have an influence.
Logs are too big to be opened but from what we could see at the start, most of the content is comes from indexing issues (due to grok parsing failure) being logged.
For now we unchecked "Collect agent logs" from the policy and all servers seem to have the "standard" 10Mb/7 files rotation.
It looks like this option has an impact on rotation.

Thanks for the help !

[ruflin]

Hi @greggailly We have seen other issues around rotation on windows and are investigating on what is causing this. Here is a related issue I recommend to follow: https://github.com/elastic/beats/issues/27853