How's working filebeat module with ingest pipeline?

Elastic Stack Beats
1

Hello, guys!

I am working with syslog + logstash + ES + kibana.
Now, i have an idea: escape logstash, add filebeat to hosts and use it with modules (i hearing that filebeat can grok events with ingest pipelines on ES node).

We don't have filebeat modules for haproxy and tomcat7, but it's not a problem. I am discovering /usr/share/filebeat/module/ and find a module structure. It's very easy to create other one module by copying filesets and editing structure of pipeline.json, adding new fields to fields.yml and other. But, i still have a quessions.

I can enable and disable my modules. I can start and stop filebeat. But. Where is the moment of PUTting new ingest pipeline from filebeat to my ES node? Maybe i skipped advanced description of this proccess in docs? What should I do in order for the filebeat to update the ingest pipeline in case it has changed in the module? How to do it manually - I know. I think that "writing a module is a good idea if it automate adding ingest pipeline to ES node".

Do you have a plan to add ingest pipeline manager to kibana? I think it can be very usefull.

Instructions for writing your personal custom module is very bad, becouse golang have a problems with building a steps, like make update. It's not useful in ubuntu 16.04. May be i am lost some dependensies? Do you have a list of it?

Thank you.

2

Hello @kingil

My reponse are inline:

We don't have filebeat modules for haproxy and tomcat7, but it's not a problem. I am discovering /usr/share/filebeat/module/ and find a module structure. It's very easy to create other one module by copying filesets and editing structure of pipeline.json, adding new fields to fields.yml and other. But, i still have a quessions.

We do have an haproxy module but you are right we don't have one for tomcat, but we do accept module contributions :slight_smile:

I can enable and disable my modules. I can start and stop filebeat. But. Where is the moment of PUTting new ingest pipeline from filebeat to my ES node? Maybe i skipped advanced description of this proccess in docs? What should I do in order for the filebeat to update the ingest pipeline in case it has changed in the module? How to do it manually - I know. I think that "writing a module is a good idea if it automate adding ingest pipeline to ES node".

Did you take a look at our developer documentation about modules? When Filebeat detect that the module have a pipeline it will take care of installing it automatically. Concerning developing an ingest pipeline we have tooling in place to help you with creating it and testing it.

Do you have a plan to add ingest pipeline manager to kibana? I think it can be very usefull.

We are discussing it, but I can't say when it will land.

Instructions for writing your personal custom module is very bad, becouse golang have a problems with building a steps, like make update. It's not useful in ubuntu 16.04. May be i am lost some dependensies? Do you have a list of it?

Lets try to fix your environment issue, did you go through these steps: Contributing to Beats | Beats Developer Guide [6.5] | Elastic

How did you install Golang from a ubuntu package?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.