Ingest Pipeline is not updated on Module Changes


(Michael Eves) #1

Hi,

I've created a new Filebeat module to parse some custom log formats. As expected on initial startup with the new module enabled Filebeat correctly created the ingest pipeline in Elasticsearch.

I've since made some changes to the module; more specifically I've added some extra processors into the ingest pipeline, however I've noticed after a rolling upgrade the ingest pipeline in Elasticsearch has not been updated.

Judging by the Filebeat logs it checks if the ingest pipeline already exists, but doesn't do a diff to see if the content is correct:

2018-06-29T19:19:53.417Z        DEBUG   [elasticsearch] elasticsearch/client.go:708     GET https://elastic:443/_ingest/pipeline/filebeat-6.3.0-golang-standard-pipeline  <nil>
2018-06-29T19:19:53.428Z        DEBUG   [modules]       fileset/pipelines.go:54 Pipeline filebeat-6.3.0-golang-standard-pipeline already loaded

Is this expected behaviour or should Filebeat be updating the ingest pipeline? If it's expected behaviour what's the recommended method to manage these pipelines?
Managing them along with other Elasticsearch cluster settings would be a solution, but it feels like it introduces a pretty big disconnect between the Filebeat module & the ingest pipeline, when those two things are really quite tightly coupled

Cheers,
Mike


Ingest Pipeline does not update on filebeat setup
(Pier-Hugues Pellerin) #2

Hello @Evesy
Running the setup subcommand should force installing the pipeline.

filebeat setup

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.