# Multi Filebeat prospector paths and Filebeat pipelines in one Logstash config file for ES/Kibana

**URL:** <https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908>\
**Category:** Elasticsearch\
**Created:** [January 11, 2019, 2:23pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908 "2019-01-11T14:23:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Datakids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/datakids/32/39622_2.png) [@Datakids](https://discuss.elastic.co/u/Datakids)\
**Post date:** [January 11, 2019, 2:23pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/1 "2019-01-11T14:23:17Z")

</div>

Hi,  
hope youre all doing fine.  
I'm quite new to the elastic stack and in learning progress.  
I play with the plugins and datasets to become more familar about this fantastic tools.  
This is my 2nd post 🙂

I want to do following but I'm not sure about the right way to do:

Logstash pipeline.yml:

- pipeline.id: test  
pipeline.workers: 1  
pipeline.batch.size: 1  
path.config: "\tmp\bin\test.conf"
- pipeline.id: anothertest  
pipeline.workers: 1  
pipeline.batch.size: 1  
path.config: "\tmp\bin\anothertest.conf"

filebeat.prospectors:

- input\_type: log  
paths:

Thanks in advance  
Regards  
Thorben

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [January 11, 2019, 2:31pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/2 "2019-01-11T14:31:50Z")

</div>

I do this with Logstash metadata

This is part of my Logstash output config

```
index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"

```

Only use one pipeline so I can't really comment on that...

You can have one Filebeat _input_ per file and add fields there that you use to generate @metadata fileds in Logstash. E.g.

```
# Adding @metadata needed for index sharding to Filebeat logs
mutate {
  copy => {
   "[fields][log_prefix]" => "[@metadata][log_prefix]"
   "[fields][log_idx]" => "[@metadata][index]"
  }
}

```

One thing to remember is that the @metadata fields are only available within Logstash...

---

<div class="post-metadata">

**Author:** ![Datakids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/datakids/32/39622_2.png) [@Datakids](https://discuss.elastic.co/u/Datakids)\
**Post date:** [January 12, 2019, 1:48pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/3 "2019-01-12T13:48:26Z")

</div>

Hi,  
ok Im not familar with this so far but I could try.  
Can you tell me where to place the second part?  
Is this also a part of the Logstash config file or do I have to place it  
inside the Filebeat.yml? And If, where to?

Thanks ans Brgds  
Thorben

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [January 14, 2019, 10:57am UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/4 "2019-01-14T10:57:41Z")

</div>

I manage Filebeat through Puppet. Using the Puppet module from Elastic I have the following config

`/etc/filebeat/filebeat.yml`  
All configs except for _inputs_ (what used to be _prospectors_) which looks like

```
...
filebeat:
  registry_file: "/var/lib/filebeat/registry"
  config.prospectors:
    enabled: true
    path: "/etc/filebeat/conf.d/*.yml"
  shutdown_timeout: '0'
  modules: []
...

```

`/etc/filebeat/conf.d/foo.yml`  
One file per input (prospector) (starts with `-`  
[More info](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html)

So `/etc/filebeat/conf.d/foo.yml` would like like

```
- type: log
  paths:
    - "/var/log/apache2/*"
  fields:
    log_prefix: dc
    log_idx: apache2
  fields_under_root: false
```

---

<div class="post-metadata">

**Author:** ![Datakids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/datakids/32/39622_2.png) [@Datakids](https://discuss.elastic.co/u/Datakids)\
**Post date:** [January 15, 2019, 9:58pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/5 "2019-01-15T21:58:35Z")

</div>

Hi,  
yeah finaly I got it and It works so far!

In filebeat.yml add as per your advice

fields:  
names: myname

and grap them in logstash.conf by  
if [fields] [names] == "myname" {  
... do something  
}  
afterwards I pass them trough the same way to create indicies based on serveral files (fields)  
if [fields] [names] == "myname" {  
elasticsearch {  
...  
}  
}

Thanks for the hint.  
Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 12, 2019, 9:58pm UTC](https://discuss.elastic.co/t/multi-filebeat-prospector-paths-and-filebeat-pipelines-in-one-logstash-config-file-for-es-kibana/163908/6 "2019-02-12T21:58:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.