Massive filtering in modules? Or in Logstash

[widhalmt (Thomas Widhalm)]

Hi,

I followed the discussion around Convert Filebeat icinga.* to ECS by webmat · Pull Request #9294 · elastic/beats · GitHub - a pull request to get the icinga module of Filebeat compatible to ECS. There were some changes that are bit confusing to me, which I try to clarify in this thread:

Implementing ECS with custom fields

Hi, We contributed the icinga module for Filebeat and started developing a Logstash pipeline for parsing Icinga logs: https://github.com/Icinga/icinga-logstash-pipeline. It seems like there are far too many different rules to implement them into the filebeat module or the ingestion pipeline so we opted for a Logstash pipeline containing all rules. I'm currently trying to line up the parsing when logs come from filebeat by reading the log on the filesystem with logs that where already preproces…

What I wanted to ask here is what you think would be the best to proceed: We started implementing a complete Logstash pipeline with all filters for Icinga 2 logs. I'm sure it's way too much filtering to be implemented in the local icinga module of Filebeat but should we think about getting it into an igest pipeline for Elasticsearch? The different syntax makes me hesitant because it was already quite a load of work to get the Logstash pipeline to the point where it is now.

What do you think?

Cheers,
Thomas