Hi,
My architecture is as follows:
Filebeat -> Logstash -> Elasticsearch
I passed a number of files to filebeat. Some of which uses some modules. These modules have been given unique var.paths in their module configuration files.
e.g. cisco.yml
- module: cisco asa: enabled: true var.paths: ["/path/to/log/*.log"]
However, I do also have some files that do not require the use of modules (i.e. JSON files). I have set the paths to these files in the filebeat.yml file.
e.g. filebeat.yml
filebeat.inputs: - type: log paths: - /path/to/log/*.json enabled: true json.keys_under_root: true fields: log_type: json-log
So my problem is, when i use the module, filebeat does not seem to have parsed the JSON files. Hence, I am wondering if i should have a separate module to manage my JSON files because i think by defining unique paths separately in the modules, the paths that are set in filebeat.input are ignore.
Also my logstash config are as follows:
e.g. in-out.conf
input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" pipeline => "%{[@metadata][pipeline]}" } } else { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } }
Please correct me if I misunderstood the use of filebeat and the use of its modules.
Any help would be very much appreciated.
Regards.