I'm playing around with filebeat and I wanted to sends logs to certain outputs in an intelligent way. That is I have a file in the /path/to/file2 which I want its logs to go to elasticsearch and another file in the /directory/to/file2 which I want to ship its logs to kafka.
I have given a try with config_dir in my main filebeat configuration file. Please find enclosed the config that I did.
filebeat.config_dir: /etc/filebeat/conf.d filebeat.prospectors: - paths: - "/path/to/file1" document_type: file1logs output.kafka: enabled: true hosts: ["ip-address-kafka:9200"] topic: topic-for-file1
filebeatFile2.yml: which is found in the directory /etc/filebeat/conf.d
filebeat.prospectors: - paths: - "/directory/to/file2" document_type: file2logs output.elasticsearch: enabled: true hosts: ["ip-address-elasticsearch:9200"] index: "file2-index"
The issue with the configuration is that output.elasticsearch is not considered. In fact it is completely ignored. That is both logs of file1 and file2 are sent to kafka.
So here is my question: can somebody now help me to achieve that please?