Hello all. My question is what is the general method for using beats with logstash if you want access to Elastic SIEM and the like?
Elastic SIEM works well when the data is gathered via beats and sent directly to Elasticsearch. For my use case I need to be able to send the data gathered from the beats over a socket, and also to elasticsearch. Since Beats don't support multiple outputs, I was wondering what the best way to do this is?
I am initially thinking of Logstash since it does support multiple outputs. However in the past Logstash didn't work with the Beats if you still want to use Elastic SIEM, I think with the issue being that their templates and ingest pipelines weren't loaded. If using Logstash is the best option, what do I need to do to circumvent this? I'm using modules so I was thinking the process would be:
Temporarily disable logstash output and post to elasticsearch the templates, as mentioned here: https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html#load-template-manually
But my question is, do I need to do this for every beat I'm using and for EACH Filebeat module template as well?
Next, load the ingest pipelines in a similar manner to Elasticsearch. Again this needs to be repeated for example Filebeat, and Suricata and Netflow modules?
Next, configure the Logstash Elasticsearch output to send to a pipeline.
Is there anything else that must be done for the data to be properly stored? I'm not sure what all the beats have to do and what of that is on the filebeat/elasticsearch side of things, so I'm really not sure.
Is this the best option? Or should I rather have 2 copies of each beat, and send one to logstash to socket and one to elasticsearch. My question with this is: what is the additional overhead of the reads from the network interface (eg with pcap or af_packet). Will this cause a significant slowdown?
Thanks for your time reading this, I'm really not sure how one would configure this conceptually and previous attempts have failed.