Filebeats for UFW logs - how to define fields?

[bvickers]

Hello.

I am trying to set up Elastic Stack to manage some typical and some custom logs.

I have Windows and Linux servers to grab logs from and I decided to use Beats to make it a little bit more secure (using logstash to control which server can push to which elasticsearch index).

The logs I want to take care of include:

• UFW logs
• Snort alerts
• typical logs like nginx, apache, etc.
• some custom logs

And I want to make them easily searchable, atomic (with separate fields and not the one huge message field). So I tried setting it all up and have realized that Beats does not include templates (fields?) for at least UFW.

UFW log entry looks like that:

Nov 3 14:49:37 XXX kernel: [UFW BLOCK] IN=eth0 OUT= MAC=XXX SRC=XXX DST=XXX LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=XXX DF PROTO=TCP SPT=XXX DPT=XXX WINDOW=XXX RES=0x00 SYN URGP=0

So I've identified two options:

• filebeat -> logstash (filter here) -> elasticsearch
• filebeat (filter here) -> logstash -> elasticsearch

I have an idea how filter (parse, map the fields, etc.) UFW logs with logstash, but can you point me to the right direction about how to filter them with filebeat?

I would be happy to see general advices and links to docs too.. Thanks in advance

[kvch (Noémi Ványi)]

Filebeat does not have as advanced filtering capabilites as Logstash does. So if you want to put together a complex filtering for your logs, I suggest you do the filtering in Logstash.
Filebeat uses processors to filter messages: https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html

To parse UFW logs the "Filebeat way" is to add a new module and a fileset. Here is a guide on how to do it: https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html
Do you mind opening and enhancement request on Github? https://github.com/elastic/beats/issues/new
I think it would be a good addition to the existing Filebeat modules. If you have time, we would appreciate if you contributed back what you have created. Let me know if you need help with creating a new module.