# Create several prospectors or several paths in one prospector ? What is the best practice?

**URL:** <https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116>\
**Category:** Beats\
**Created:** [September 11, 2018, 12:12pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116 "2018-09-11T12:12:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [September 11, 2018, 12:12pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/1 "2018-09-11T12:12:03Z")

</div>

Hello.

I have severals directory to listen thanks to filebeat on my server :  
/data/EDT/1/batchsefluid/files/logs/  
/data/EDT/2/batchsefluid/files/logs/  
/data/EDT/3/batchsefluid/files/logs/  
/data/EDT/4/batchsefluid/files/logs/  
/data/EDT/5/batchsefluid/files/logs/  
.  
.  
until 10.

And what is the best way to do it with filebeat :

Create 1 prospector and add 10 "paths"

Or create 10 prospectors with 1 "path" ?

Obviously i'll add filetypes to these directory to filter them later with Kibana.

Thank

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [September 11, 2018, 2:09pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/2 "2018-09-11T14:09:06Z")

</div>

If you don't add custom fields to the prospector, one is enough.

---

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [September 11, 2018, 2:25pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/3 "2018-09-11T14:25:47Z")

</div>

OK so, for you If I don't add custom fields, my filebeat.conf could looks like :

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /data/EDT/1/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number1

- type: log
  paths:
    - /data/EDT/2/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number2

- type: log
  paths:
    - /data/EDT/3/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number3

```

And then I filter later with my filetype to see only a directory into Kibana

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [September 11, 2018, 2:33pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/5 "2018-09-11T14:33:43Z")

</div>

@dyl Yes that will work, you could also generalize your configuration. Using the dissect processor to extract the `2` in the `source` and only have 1 prospector.

---

<div class="post-metadata">

**Author:** ![dyl](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@dyl](https://discuss.elastic.co/u/dyl)\
**Post date:** [September 12, 2018, 6:09am UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/6 "2018-09-12T06:09:09Z")

</div>

Ok so I could do :

```auto
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /data/EDT/1/batchsefluid/files/logs/*.log
    - /data/EDT/2/batchsefluid/files/logs/*.log
    - /data/EDT/3/batchsefluid/files/logs/*.log

```

And then I parse a `source` field to extract 1, 2 and 3 that's it ?

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [September 12, 2018, 12:06pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/7 "2018-09-12T12:06:50Z")

</div>

Yes using the [dissect processor](https://www.elastic.co/guide/en/beats/filebeat/master/dissect.html), I believe the following tokenizer would work:

```auto
/data/EDT/%{filetype}/%{?rest}

```

And you can access the value with `%{[dissect.filetype]}`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2018, 2:06pm UTC](https://discuss.elastic.co/t/create-several-prospectors-or-several-paths-in-one-prospector-what-is-the-best-practice/148116/8 "2018-10-10T14:06:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.