Hi Everyone, I am trying to configure filebeat to output to a totally different index in case a certain condition is met as shown below but also to use a different index template:
- type: log
enabled: true
paths:
- N:/logs/service1/logstash/audi*.log
json:
keys_under_root: true
add_error_key: true
fields:
service: "service1-audit"
department: "departmentname"
project: "service1"
audit: true
- type: log
enabled: true
paths:
- N:/logs/service1/logstash/logstas*.log
json:
keys_under_root: true
add_error_key: true
fields:
service: "service1"
department: "departmentname"
project: "service1"
setup:
template:
enabled: true
name: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
pattern: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
ilm.enabled: false
output.elasticsearch:
hosts: ["https://host1:9200","https://host2:9201","https://host3:9202"]
ssl.certificate_authorities: ['C:\filebeat\certs\ca.crt']
protocol: https
username: "filebeat"
password: "strong-password"
data_stream: true
indices:
- index: "secure-%{[fields.department]:unknown}-%{[fields.service]:unknown}-audit"
when.equals:
fields.audit: "true"
- index: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
not:
when.equals:
fields.audit: "true"
bulk_max_size: ${buffer_size}
workers: 2
compression_level: 3
The logs are being collected from the paths specified correctly but although the field.audit is populated the logs are still being written to the datastream called: comp-departmentname-service1.
I suspect that this is happening as although the destination indices are specified with proper conditions, the template is always refering to the service which does not have the audit field.
Can you please let me know how I can tackle such issue please or if I am on the wrong track here?
Thanks
Luke