So recently we've started out our journey with Elastic 7.4.x and what a joy it has been. We wanted to implement ECS as it seems the right way forward. However Beats, logstash, kibana and elastic is not a combination made easy.
Let me start of by saying that a common model and common examples are a good way to go. However if you run anything else then the default elastic cluster things are not so easy. I run a combination of logstash for ingestion and kafka for a persistent queue. Therefore my beats don't have direct access to either kibana or elastic. It's neatly sealed behind a firewall/forward proxy with restricted access from my office.
Immediately you run into problems:
How do you get dashboards from beats to kibana? Simple answer: you don't. And if you do, you will get a zillion dashboards in your default space, cluttering up kibana. I ended up setting up a testing cluster to export the dashboards to, selecting the dashboards i wanted and importing it into my production kibana. Can't we at least have a json file downloadable somehwere that I can import into a different space so it doesn't clutter up my default space?
How do you get the ECS templates from filebeat/metricbeat into elastic? Luckily there is an export command, but the templates are different!!! For something as common as ECS, the templates are different. I would just love to have one ECS template and register all my beats to that template. Now I have to juggle different templates again.
I run haproxy and filebeat has a haproxy module. But that module does nothing for the haproxy/url/httpo specific fields. All you get is a common syslog message. You need an elastic ingest node for that, but I run logstash. So I ended up rewriting the entire ingest stuff into logstash compatible stuff.
How do you add fields to the common schema? The github page does not explain how to add fields or which scripts to run to create a new template. Also the haproxy fields are nowhere to be found in the git. If i start messing with the default ECS how do I add the haproxy fields? (where are they defined?) I need to have a few application specific fields (in my case arcgis stuff) that I want to add to the ECS.
All in all, my journey hasn't been without problems. The first part is a bit of a rant, but I needed to get it out of my system. The second part is my real question. Where can I find the haproxy defined fields and how do I compile my own template.