I am trying to set up Elastic Stack to manage some typical and some custom logs.
I have Windows and Linux servers to grab logs from and I decided to use Beats to make it a little bit more secure (using logstash to control which server can push to which elasticsearch index).
The logs I want to take care of include:
- UFW logs
- Snort alerts
- typical logs like nginx, apache, etc.
- some custom logs
And I want to make them easily searchable, atomic (with separate fields and not the one huge message field). So I tried setting it all up and have realized that Beats does not include templates (fields?) for at least UFW.
UFW log entry looks like that:
Nov 3 14:49:37 XXX kernel: [UFW BLOCK] IN=eth0 OUT= MAC=XXX SRC=XXX DST=XXX LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=XXX DF PROTO=TCP SPT=XXX DPT=XXX WINDOW=XXX RES=0x00 SYN URGP=0
So I've identified two options:
- filebeat -> logstash (filter here) -> elasticsearch
- filebeat (filter here) -> logstash -> elasticsearch
I have an idea how filter (parse, map the fields, etc.) UFW logs with logstash, but can you point me to the right direction about how to filter them with filebeat?
I would be happy to see general advices and links to docs too.. Thanks in advance