# Manual activation of modules

**URL:** https://discuss.elastic.co/t/manual-activation-of-modules/223382
**Category:** Beats
**Tags:** filebeat
**Created:** [March 12, 2020, 4:59pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382 "2020-03-12T16:59:25Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2020, 4:59pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382/1 "2020-03-12T16:59:25Z")

</div>

I'm trying to understand howto manually pre-setup ingest pipelines for various wanted filebeat modules and configure modules to use such pipelines.

Eg let's look at system module, this has two parts syslog + audit logs, each has under /usr/share/filebeat/module/system a manifest.yml referencing a config/\*.yml file and ingest/pipeline.json, would this automatical get loaded into my elasticsearch output cluster by running:

```
/usr/bin/filebeat modules enable system

```

Or need I pre-define required pipeline(s) and if so how to name such and reference them under each sub-module input config or are such pipellines simply just processed locally in filebeat before sending any output and thus have no need to named 🙂

/usr/share/filebeat/module/system/auth/ manifest.yml, config/auth.yml, ingest/pipeline.json:

 ![Screenshot 2020-03-12 at 17.46.08](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b1032a9ff567a1f5a5b89e8402216e26e6389511.png)

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 16, 2020, 3:13pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382/2 "2020-03-16T15:13:28Z")

</div>

So it seems that filebeat wants/needs the 'manage\_ingest\_pipeline' privilege (a privilege I can't find under Roles in kibana btw) to install any needed pipelines, sigh 😕

So I first launched a filebeat with my super users' privilege and it then installed pipelines as expected, but reverting to launching with our less-privileged ingest user we're still seeing:

```
2020-03-16T15:51:53.757+0100	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://<redacted>:<redacted>)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [<redacted>]"},"status":403}  

```

even though I've set:

```
filebeat.overwrite_pipelines: false
setup.ilm.enabled: false
setup.template.enabled: false

```

Totally prefer to pre-setup everything needed before rolling out beats to multiple end points and thus run beats with least required privileges as discussed [here](https://github.com/elastic/beats/issues/10241) filebeat is an as\* for this.

Any hints on below Qs much appreciated...

What's recommend to overcome this?

Could filebeat also still have a defined general pipeline (output.elasticsearch.pipeline) that always will be executed regardless of possible separate modules' pipelines?

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [March 18, 2020, 2:19pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382/3 "2020-03-18T14:19:27Z")

</div>

It is not possible to disable pipeline loading if modules are enabled. It is a safety measure to make sure if the user has forgotten to set up the pipelines, the events can be still parsed properly.

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 18, 2020, 2:33pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382/4 "2020-03-18T14:33:07Z")

</div>

Don't want to disable pipeline either 🙂

Seems that by following [this page](https://www.elastic.co/guide/en/beats/filebeat/current/feature-roles.html#privileges-to-publish-events) and creating a filebeat\_writer role and assigning this to my ingest user filebeat can run without complaining though granting manage\_ingest\_pipeline cluster wide seems not to be applying-least-privileges principal ImHO 😕

Reading the doc it's should be possible to define output.elasticsearch.pipeline as well as the automatical module defined pipeline, question is if documents will parse through both/all pipelines?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 15, 2020, 2:33pm UTC](https://discuss.elastic.co/t/manual-activation-of-modules/223382/5 "2020-04-15T14:33:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.