My goal is to get the exact number of logs along the path specified in auditbeat in elastic to build a schedule for receiving logs by day in kibana.
The problem is that I get several duplicates of the same log in elastic. Auditbeat manages to track changes in the log file and send several messages through it.
The multiline log file consists of a request and a response to a request, after receiving a response, the state of the file does not change, the response comes in a few fractions of seconds:
2021-12-09 16:52:24.6435|INFO|logfile|Запрос:
2021-12-09 16:52:24.6747|INFO|logfile|Ответ:
Help to adjust the frequency of scanning files to avoid duplicates.
Or maybe there is a way how to attach the generated hash of the file as its id in elastic?
My auditbeat config:
auditbeat.modules:
- module: file_integrity
paths:
- C:/xml_logs/GT/xml/
scan_at_start: true
recursive: true
setup.template.settings:
index.number_of_shards: 1
tags: ["xml"]
output.logstash:
hosts: ["10.1.1.4:5044"]
logging:
to_files: true
files:
path: C:/ProgramData/auditbeat/Logs
level: debug