Filebeat Performance

[ori.rubinfeld]

Hi,

My filebeat config file has 5 prospectors.
Each one access a network path on one of five application servers.

app_server_01, d:\app_server\logs\app_server_01.log
app_server_02, d:\app_server\logs\app_server_02.log
app_server_03, d:\app_server\logs\app_server_03.log
app_server_04, d:\app_server\logs\app_server_04.log
app_server_05, d:\app_server\logs\app_server_05.log

Logs are being rotated.

The Filebeat sends the data to ingest node in Elastic.

While accessing the data using Kibana.
It seems that data is being received in Realtime from app_server_01, but from app_server_03, there is a long delay, like for 30 minutes and data has not been received yet.

After a while, the data does get indexed in Elastic Search.

I need to have it all in as close as it can be to Real Time.

What can be checked for the Configuration ?
What can cause such a delay on one machine, and no delay on the other (Using same FB, different prospectors accessing the files) ?

Thanks,

Ori

[andrewkroh]

Please share your Filebeat config, version, and OS.

If you are reading the files over a network share I would think this is most likely the cause. See Can’t read log files from network volumes?.

[ori.rubinfeld]

Hi,

Apparently it is a Log4J issue on windows machines.
The Modification data is not being updated while the file is being populated, but randomly.

This is a very serious issue, as we can not provide a Real Time indexing of the logs data.

Ori