Transmission of logs in real time mode

Elastic Stack Elasticsearch
1

Hello everybody

Please tell me what the problem might be

I have a mail server on which the filebeat agent is installed, it transfers data to another server on which logstash and elastic are installed

I randomly display data with a delay of more than 7 seconds

Записати
Записати1449×217 32.2 KB

As you can see in the screenshot, the recording time of the event on the mail server differs from the recording time in the elastic itself by 7 seconds

What could this be related to?

All VMs are on the same network in the same VLAN on the same host in VMware

2

This could be a number of things:

  • buffering in filebeat
  • buffering in logstash
  • etc

I would recommend you to parse the events using a pipeline (either logstash or elasticsearch) and use the date filter/processor to set the @timestamp to the one in the log record.

This will make sure the records are displayed in the correct time moment.

3

And could you provide more practical recommendations. What parameters to pay attention to?

Thank you!