Manual activation of modules

[stefws (Steffen Winther Sørensen)]

I'm trying to understand howto manually pre-setup ingest pipelines for various wanted filebeat modules and configure modules to use such pipelines.

Eg let's look at system module, this has two parts syslog + audit logs, each has under /usr/share/filebeat/module/system a manifest.yml referencing a config/*.yml file and ingest/pipeline.json, would this automatical get loaded into my elasticsearch output cluster by running:

/usr/bin/filebeat modules enable system

Or need I pre-define required pipeline(s) and if so how to name such and reference them under each sub-module input config or are such pipellines simply just processed locally in filebeat before sending any output and thus have no need to named

/usr/share/filebeat/module/system/auth/ manifest.yml, config/auth.yml, ingest/pipeline.json:

[stefws (Steffen Winther Sørensen)]

So it seems that filebeat wants/needs the 'manage_ingest_pipeline' privilege (a privilege I can't find under Roles in kibana btw) to install any needed pipelines, sigh

So I first launched a filebeat with my super users' privilege and it then installed pipelines as expected, but reverting to launching with our less-privileged ingest user we're still seeing:

2020-03-16T15:51:53.757+0100	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://<redacted>:<redacted>)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"},"status":403}  

even though I've set:

filebeat.overwrite_pipelines: false
setup.ilm.enabled: false
setup.template.enabled: false

Totally prefer to pre-setup everything needed before rolling out beats to multiple end points and thus run beats with least required privileges as discussed here filebeat is an as* for this.

Any hints on below Qs much appreciated...

What's recommend to overcome this?

Could filebeat also still have a defined general pipeline (output.elasticsearch.pipeline) that always will be executed regardless of possible separate modules' pipelines?

[kvch (Noémi Ványi)]

It is not possible to disable pipeline loading if modules are enabled. It is a safety measure to make sure if the user has forgotten to set up the pipelines, the events can be still parsed properly.

[stefws (Steffen Winther Sørensen)]

Don't want to disable pipeline either

Seems that by following this page and creating a filebeat_writer role and assigning this to my ingest user filebeat can run without complaining though granting manage_ingest_pipeline cluster wide seems not to be applying-least-privileges principal ImHO

Reading the doc it's should be possible to define output.elasticsearch.pipeline as well as the automatical module defined pipeline, question is if documents will parse through both/all pipelines?