Filebeat putting too much load on the system. Seems to be related to gc.log

[Webtrend_Inc]

On our elastic 7.7.1 stack, when we enabled our elastic log ingestion via filebeat, we saw a significant load on our data and ingestion nodes.

Our log path parameter was defined as /var/log/elasticsearch/*.json and all other parameters were at their respective default values.

After much trial and error, we determined that it was most likely being caused by the presence of old garbage collector log files which were in the format gc.log.nn (gc.log.01, gc.log.02 etc)

We tried to prevent filebeat from reading these files by adding regex to exclude these files

exclude_files: ['.gz$', '^gc']

but that did not have any impact. When we deleted those gc logs, the problem disappeared.

So I am wondering why is filebeat parsing those gc log files (assuming our assessment is true) when there is no directive in yaml config to do so.

[Webtrend_Inc]

I forgot to add this to my earlier post.

When I look at the filebeat index, it already has 48 million documents in it and almost all of them are from ingesting gc.log files from various nodes.

So, my original question remains: How do I prevent filebeat from ingesting gc.log ?

[Webtrend_Inc]

Answering my own question: I had the elasticsearch module enabled and that is why it was parsing gc logs