Hi all,
So recently we've started out our journey with Elastic 7.4.x and what a joy it has been. We wanted to implement ECS as it seems the right way forward. However Beats, logstash, kibana and elastic is not a combination made easy.
Let me start of by saying that a common model and common examples are a good way to go. However if you run anything else then the default elastic cluster things are not so easy. I run a combination of logstash for ingestion and kafka for a persistent queue. Therefore my beats don't have direct access to either kibana or elastic. It's neatly sealed behind a firewall/forward proxy with restricted access from my office.
Immediately you run into problems:
-
How do you get dashboards from beats to kibana? Simple answer: you don't. And if you do, you will get a zillion dashboards in your default space, cluttering up kibana. I ended up setting up a testing cluster to export the dashboards to, selecting the dashboards i wanted and importing it into my production kibana. Can't we at least have a json file downloadable somehwere that I can import into a different space so it doesn't clutter up my default space?
-
How do you get the ECS templates from filebeat/metricbeat into elastic? Luckily there is an export command, but the templates are different!!! For something as common as ECS, the templates are different. I would just love to have one ECS template and register all my beats to that template. Now I have to juggle different templates again.
-
I run haproxy and filebeat has a haproxy module. But that module does nothing for the haproxy/url/httpo specific fields. All you get is a common syslog message. You need an elastic ingest node for that, but I run logstash. So I ended up rewriting the entire ingest stuff into logstash compatible stuff.
-
How do you add fields to the common schema? The github page does not explain how to add fields or which scripts to run to create a new template. Also the haproxy fields are nowhere to be found in the git. If i start messing with the default ECS how do I add the haproxy fields? (where are they defined?) I need to have a few application specific fields (in my case arcgis stuff) that I want to add to the ECS.
All in all, my journey hasn't been without problems. The first part is a bit of a rant, but I needed to get it out of my system. The second part is my real question. Where can I find the haproxy defined fields and how do I compile my own template.