Create several prospectors or several paths in one prospector ? What is the best practice?

[dyl (bus)]

Hello.

I have severals directory to listen thanks to filebeat on my server :
/data/EDT/1/batchsefluid/files/logs/
/data/EDT/2/batchsefluid/files/logs/
/data/EDT/3/batchsefluid/files/logs/
/data/EDT/4/batchsefluid/files/logs/
/data/EDT/5/batchsefluid/files/logs/
.
.
until 10.

And what is the best way to do it with filebeat :

Create 1 prospector and add 10 "paths"

Or create 10 prospectors with 1 "path" ?

Obviously i'll add filetypes to these directory to filter them later with Kibana.

Thank

[pierhugues (Pier-Hugues Pellerin)]

If you don't add custom fields to the prospector, one is enough.

[dyl (bus)]

OK so, for you If I don't add custom fields, my filebeat.conf could looks like :

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    -  /data/EDT/1/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number1

- type: log
  paths:
    -  /data/EDT/2/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number2

- type: log
  paths:
    -  /data/EDT/3/batchsefluid/files/logs/*.log
  fields_under_root: true
  fields:
   filetype: number3

And then I filter later with my filetype to see only a directory into Kibana

[pierhugues (Pier-Hugues Pellerin)]

@dyl Yes that will work, you could also generalize your configuration. Using the dissect processor to extract the 2 in the source and only have 1 prospector.

[dyl (bus)]

Ok so I could do :

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    -  /data/EDT/1/batchsefluid/files/logs/*.log
    -  /data/EDT/2/batchsefluid/files/logs/*.log
    -  /data/EDT/3/batchsefluid/files/logs/*.log

And then I parse a source field to extract 1, 2 and 3 that's it ?

[pierhugues (Pier-Hugues Pellerin)]

Yes using the dissect processor, I believe the following tokenizer would work:

/data/EDT/%{filetype}/%{?rest}

And you can access the value with %{[dissect.filetype]}