Audit logs are not rotated correctly when the elasticsearch module is enabled

[gerarddp]

When enabling the audit log parsing with the elasticsearch module for filbeat, the audit logs are not rotate correctly because filebeat keeps the files open. That also causes the disk to get full quickly because of the amount of audit logs.

Is there a way to keep using the module without this issues?

[kvch (Noémi Ványi)]

The issue is not coming from using the module. The root cause is that the output is processing the events from Filebeat too slowly. Do you have any errors in the logs of the output?
Also, you can set the appropiate close_* option, so Filebeat closes files: https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html#close-options
As you are using a module, you need to set it under input:

#---------------------------- Elasticsearch Module ----------------------------
- module: elasticsearch
  server:
    enabled: true

    input:
      close_removed: true

Note that I am not suggesting that close_removed is something you need to enable. The options to use/enable depend on your logs and logrotation.

[gerarddp]

I've enabled close_removed but the issue is not fixed. Also the issue is with the audit part, not server. All the others work correctly, but there's not as many logs as the audit.

I've tried searching for any errors, but I cannot see any. The output is elasticsearch itself, so maybe there's a loop issue?

Maybe this will help (with close_removed: true)

#lsof -p 32060
COMMAND    PID USER   FD      TYPE             DEVICE  SIZE/OFF    NODE NAME
[...]
filebeat 32060 root    6r      REG              254,1 268435773  262779 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root    7r      REG              254,1 268435627  262783 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root    8r      REG              254,1 268435670  262784 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root    9r      REG              254,1 268435549  262785 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   10r      REG              254,1 268435673  263014 /var/log/elasticsearch/logging_audit-2019-08-05-3.json
filebeat 32060 root   11r      REG              254,1 268435528  262786 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   12r      REG              254,1 268435505  262926 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   13r      REG              254,1  40754471  262789 /var/log/elasticsearch/gc.log
filebeat 32060 root   14r      REG              254,1  40754471  262789 /var/log/elasticsearch/gc.log
filebeat 32060 root   15r      REG              254,1 268435663  262928 /var/log/elasticsearch/igp-logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   16r      REG              254,1 268435496  262976 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   17r      REG              254,1 268435484  263115 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   18r      REG              254,1 268435845  263565 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   19r      REG              254,1 268435862  263568 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   20r      REG              254,1 268435533  263573 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   21r      REG              254,1 268435853  263574 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   22r      REG              254,1 268435705  263575 /var/log/elasticsearch/logging_audit-2019-08-05-1.json (deleted)
filebeat 32060 root   23r      REG              254,1 268435801  263576 /var/log/elasticsearch/logging_audit-2019-08-05-1.json
filebeat 32060 root   24r      REG              254,1 268435853  263577 /var/log/elasticsearch/logging_audit-2019-08-05-2.json
filebeat 32060 root   25r      REG              254,1 268435847  263578 /var/log/elasticsearch/logging_audit-2019-08-05-4.json
filebeat 32060 root   26r      REG              254,1 268435539  263579 /var/log/elasticsearch/logging_audit-2019-08-05-5.json
filebeat 32060 root   27r      REG              254,1  68181742  262782 /var/log/elasticsearch/logging_audit.json

[kvch (Noémi Ványi)]

What do you use to rotate your logs? Could you share its configuration?

[gerarddp]

We didn't change any configuration, the rotation is done by elasticsearch. We just enabled audit logs.