# Ingest Pipeline with filebeat not working
**URL:** https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357
**Category:** Kibana
**Tags:** ingest-pipeline
**Created:** [February 23, 2023, 5:58pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357 "2023-02-23T17:58:46Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@heisenberg93](https://discuss.elastic.co/u/heisenberg93)
#### Post date: [February 23, 2023, 5:58pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/1 "2023-02-23T17:58:46Z")
Hi,
We have a few Linux hosts which monitor each other with Pacemaker. Here we would like to evaluate the pacemaker.log for this. With Filebeat the log arrives, but the message is not yet analyzable. Therefore I thought with an own ingest pipeline this would be best possible.
Each Message in our pacemaker log, looks like this:
`Feb 23 18:48:33 host pacemaker-schedulerd[1669] (pcmk__log_transition_summary) notice: Calculated transition 3386, saving inputs in /var/lib/pacemaker/pengine/pe-input-374.bz2`
Now I created an ingest pipeline, with the following information:
`PUT _ingest/pipeline/filebeat-8.6.2-pacemaker { "description": "Filebeat pacemaker pipeline", "processors": [{ "set": { "field": "event.ingested", "value": "{{_ingest.timestamp}}" } }, { "gsub": { "field": "message", "pattern": "\\t", "replacement": "" } }, { "grok": { "field": "message", "patterns": [ "%{MONTH} %{MONTHDAY} %{TIME} %{HOSTNAME:hostname} %{DATA:process_name}\\[%{INT:process_id}\\] \\(%{WORD:log_type}\\) %{WORD:status}: %{GREEDYDATA:error_message}" ] } }, { "rename": { "field": "hostname", "target_field": "fileserver.host" } }, { "rename": { "field": "process_name", "target_field": "fileserver.process" } }, { "rename": { "field": "status", "target_field": "fileserver.status" } }, { "rename": { "field": "error_message", "target_field": "fileserver.error" } }, { "rename": { "field": "log_type", "target_field": "fileserver.log_type" } }, { "rename": { "field": "process_id", "target_field": "fileserver.process_id" } } ], "on_failure": [{ "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } }] }`
The good news here, with the json file from an log entry, it works and the message can be decoded. But if im integrate it into my host in the filebeat.yml:
- type: filestream
id: my-filestream-id
enabled: true
And with this setting I became no data and the filebeat process on the host is responding with success. Could anybody help me here?
Thanks in advance
---
### Author:  [@eMitch](https://discuss.elastic.co/u/eMitch)
#### Post date: [February 23, 2023, 7:28pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/2 "2023-02-23T19:28:33Z")
Hi @heisenberg93 and welcome to the community!
the formatting of your config files makes it a little difficult to understand entirely, but from what i can tell you may need to move the pipeline from the `path` section to the `output` section. Take a look at [the docs here for reference](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html).
---
### Author:  [@heisenberg93](https://discuss.elastic.co/u/heisenberg93)
#### Post date: [February 23, 2023, 9:20pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/3 "2023-02-23T21:20:25Z")
Okay, perfect, this works. But I got another service which generates logs and the message field got a different structure. Whats the best way here? Should I look out for a better ingest pipeline or is it possible to give every log directory an own ingest pipeline?
---
### Author:  [@heisenberg93](https://discuss.elastic.co/u/heisenberg93)
#### Post date: [February 24, 2023, 8:53am UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/4 "2023-02-24T08:53:01Z")
So I collect many data and the ingest pipeline is active even for my modules. But I only want it for the type filestream. Is there no chance?
---
### Author:  [@eMitch](https://discuss.elastic.co/u/eMitch)
#### Post date: [February 24, 2023, 2:28pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/5 "2023-02-24T14:28:07Z")
You _can_ use a pipeline on the [input of the filestream](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_pipeline_6) if you want to specify different pipelines for each input. Also, if you have an input and an output pipeline specified, then the input pipeline is used.
Try moving the `pipeline` between the `id` and the `enabled` tags, something like this (as an example):
```auto
filebeat.inputs:
- type: filestream
id: my-filestream-id
pipeline: filebeat-8.6.2-pacemaker
enabled: true
paths:
- /var/log/pacemaker/*.log
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 24, 2023, 2:28pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-filebeat-not-working/326357/6 "2023-03-24T14:28:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.