I am working with syslog + logstash + ES + kibana.
Now, i have an idea: escape logstash, add filebeat to hosts and use it with modules (i hearing that filebeat can grok events with ingest pipelines on ES node).
We don't have filebeat modules for haproxy and tomcat7, but it's not a problem. I am discovering /usr/share/filebeat/module/ and find a module structure. It's very easy to create other one module by copying filesets and editing structure of pipeline.json, adding new fields to fields.yml and other. But, i still have a quessions.
I can enable and disable my modules. I can start and stop filebeat. But. Where is the moment of PUTting new ingest pipeline from filebeat to my ES node? Maybe i skipped advanced description of this proccess in docs? What should I do in order for the filebeat to update the ingest pipeline in case it has changed in the module? How to do it manually - I know. I think that "writing a module is a good idea if it automate adding ingest pipeline to ES node".
Do you have a plan to add ingest pipeline manager to kibana? I think it can be very usefull.
Instructions for writing your personal custom module is very bad, becouse golang have a problems with building a steps, like make update. It's not useful in ubuntu 16.04. May be i am lost some dependensies? Do you have a list of it?