Hello Everyone,
While debugging the issue with TCP input with logstash, I have introduced Rsyslog as an intermediate between SIEM and Logstash as a temporary workaround.
Please refer to below link -
SIEM -> Rsyslog (TCP) -> Logstash (TCP)
Even though this approach is working, there is significant amount of data being dropped between Rsyslog and Logstash.
For instance, If SIEM is sending 1,000,000 records, Rsyslog is capturing 997,000 on average but Logstash is capturing only 500,000 records
I have tried persistent queues as well
Please refer to below logstash.yml
path.data: /var/lib/logstash
queue.type: persisted
path.queue: /home/logstash
queue.max_bytes: 128mb
dead_letter_queue.enable: true
path.dead_letter_queue: /home/logstash/dead_queue
path.logs: /var/log/logstash
config.reload.automatic: true
config.reload.interval: 1s
Kindly let me know if you have any suggestions to try out
Thank You !!!