Elastic Agent filling up disk space with logs, disaster

Elastic Security Endpoint Security
1

We had a meltdown on one of our servers, where the elastic agent filled up the disk space with logs (36GB of text logfiles, in 2 months), from a trial we had on the Elastic Cloud.

I know I am to blame for installing an experimental beta on a production server. Anyone knows how this happened? (Trial ended, so elasticsearch cluster went offline, possible cause?)

Anyone knows how to configure on how to prevent this from happening again?

2

Hi @Achilleas Thanks for trying out Agent. I'm sorry to hear that happened to you. Could you please share the path(s) to the files that filled up your disk to help us narrow down where the problem is?

3

Agents will probably log a lot of messages if they can't send events, so when the cluster went offline, the messages started.

I've noticed that Elastic doesn't send logrotate configs for their logs. If you're LInux, I'd suggest you always define logrotate policy for anything that produces log files.

If you're Windows, I'm sorry :slight_smile: ( I don't know the alternative for logrotate there....)

4

I am afraid I deleted them ASAP as I really had to do damage control.
I do remember they were under "C:\Program Files\Elastic....\logs"

I guess there should be a limit to log retention in case of ELK unreachable...

5

Thank you for that path, even though it was only from memory and a bit incomplete. I was not able to reproduce that path filling up the disk under normal circumstances. However, when I mimicked a possible behavior some other antivirus or backup software on your system may have I was able to get that path to fill up.

We'll put in place mitigations so that if it this was caused by another application on your computer that that it won't lead to your hard drive filling up in the future.

6

Thank you for looking into this. No other security software was present at the server. To my eyes, it is obvious that these accumulating logs were the logs that could not be sent to elasticsearch. So is there a limit implemented there by default, or are they stacking up indefinately?

7

I missed before that there are two logs directories under c:\Program Files\Elastic. The one I was referring to is under c:\Program Files\Elastic\Endpoint. There is another under c:\Program Files\Elastic\Agent.

Both have max size caps for the files under the directory. The Endpoint one should never grow above 125MB (a 100MB limit , with log files up to 25MB files in size leading to possibly 125MB of data at times). The Agent one 70MB (up to 7 10MB files, configurable with logging.files.keepfiles option).

So, this is a long winded way of saying that whatever you saw was a bug. If you see it reproduce again please reach out and let us know. Key to us understanding where the bug is will be knowing a full directory listing (including file sizes) for everything under c:\Program Files\Elastic.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.