My custom module ignores pipeline

[gaetanoziri]

Hi,

I builded a custom module for my application following this guide.

After the setup of this module i can see the messages form my module in kabana, but they are not parsed with my pipeline.

Below the steps that i did:

1. setup filebeat .\filebeat.exe setup -e

2. enable the module .\filebeat.exe modules enable <mymodule>

3. setup the module .\filebeat.exe setup --modules <mymodule> -e

Note, I tested my pipeline using API.

There are others steps to do?

Thanks,
Gaetano

UPDATE:
If I run the below commands it seems work correctly:

1. stop the filebeat service Stop-Service filebeat

2. disable the module .\filebeat.exe modules disable <mymodule>

3. run this command: .\filebeat.exe -modules=<mymodule> -setup -e

This behavior is really strange because the service is stopped, the module is disabled but in kibana i can see the messages correctly parsed.

[steffens (Steffen Siering)]

When running filebeat setup, you should also pass --pipelines in order to install the ingest pipelines.

The later command not only setups, but also runs filebeat. This might have setup the module once the first event has been seen.

[gaetanoziri]

@steffens

When running filebeat setup , you should also pass --pipelines in order to install the ingest pipelines.

The pipelines are installed, i can see the ingest pipelines if i run this http request GET _ingest/pipeline/.

However, i tryed to run .\filebeat.exe setup --modules <mymodule> -e --pipelines but the parsing still doesn't work.

[steffens (Steffen Siering)]

Now you got me confused. So you say the module is enabled, ingest node pipeline is correctly installed, but filebeat is not using the pipeline you did create?

Can you share your module definition, configuration and filebeat logs?

[gaetanoziri]

@steffens

Now you got me confused. So you say the module is enabled, ingest node pipeline is correctly installed, but filebeat is not using the pipeline you did create?

Yes.

@steffens

Can you share your module definition, configuration and filebeat logs?

You can see this commit.

In the logs i see only this two lines:

2018-12-06T17:20:21.172+0100	INFO	instance/beat.go:608	Home path: [C:\temp\filebeat-6.4.3-windows-x86_64] Config path: [C:\temp\filebeat-6.4.3-windows-x86_64] Data path: [C:\temp\filebeat-6.4.3-windows-x86_64\data] Logs path: [C:\temp\filebeat-6.4.3-windows-x86_64\logs]
2018-12-06T17:20:21.212+0100	INFO	instance/beat.go:615	Beat UUID: 0b0c66ab-f277-4a53-bac4-feb8806d976c