Hi,
I've created a new Filebeat module to parse some custom log formats. As expected on initial startup with the new module enabled Filebeat correctly created the ingest pipeline in Elasticsearch.
I've since made some changes to the module; more specifically I've added some extra processors into the ingest pipeline, however I've noticed after a rolling upgrade the ingest pipeline in Elasticsearch has not been updated.
Judging by the Filebeat logs it checks if the ingest pipeline already exists, but doesn't do a diff to see if the content is correct:
2018-06-29T19:19:53.417Z DEBUG [elasticsearch] elasticsearch/client.go:708 GET https://elastic:443/_ingest/pipeline/filebeat-6.3.0-golang-standard-pipeline <nil>
2018-06-29T19:19:53.428Z DEBUG [modules] fileset/pipelines.go:54 Pipeline filebeat-6.3.0-golang-standard-pipeline already loaded
Is this expected behaviour or should Filebeat be updating the ingest pipeline? If it's expected behaviour what's the recommended method to manage these pipelines?
Managing them along with other Elasticsearch cluster settings would be a solution, but it feels like it introduces a pretty big disconnect between the Filebeat module & the ingest pipeline, when those two things are really quite tightly coupled
Cheers,
Mike