When should we use a single prospector and when multiple?
Like I can include multiple files in a single prospector. What are the pros and cons of using multiple prospectors?
Each prospector offers lots of different configuration options like include_lines, exclude_lines, fields etc: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#_exclude_lines
If these are all the same for your log files, you can use one prospector. If you need for example different fields for your log files, you need to split them up into different prospectors.
I hope this helps.
1 Like
Then how do you manage which config file manages which log? Suppose I make two propectors, one parses app log and another wire log. The config file for both will be different. But they both are being received at the same port.
How do I specify that which config file manage which log. Can I attach a tag or something?
What do you mean by which config file? Prospector is an array.
What you are looking for is probably fields: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#configuration-fields
If I add a field in my beat, say, type_of_field=1,
And I want to use this field to determine, which of the config files of logstash should it go to.
Suppose I have a config file in logstash for parsing "a" types of logs and another for parsing "b" types of logs., how do I send logs to the appropriate file?
Wrap your Logstash filters and outputs in conditionals.
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html
Can I set different multiline conditions for different paths specified in my filebeat.yml file?
The documentation page I linked to contains many examples in the Conditionals section.
In my filter.yml file, when I specify different multiline conditions for different propectors, only one of them is running, rest of them are not. What can possibly be the error?
Like only the path which is defined in the end is sending logs to the logstash, the rest are not.
Can you share your config? I assume you missed the - between each prospector to make it an array.
Ya, that's exactly what it was. I figured it out. Actually I was using different types of logs within the same prospector. Thanks.
As I have seen this issue in the past I'm curious what brought you to the conclusion it would work without the - ? Are the docs not clear or some examples we should improve?
No, it's okay. The documentation is fine. It was just a mistake that I missed out the '-'.