Index and template conditions

[luke.camilleri]

Hi Everyone, I am trying to configure filebeat to output to a totally different index in case a certain condition is met as shown below but also to use a different index template:

- type: log
  enabled: true
  paths:
    - N:/logs/service1/logstash/audi*.log
  json:
    keys_under_root: true
    add_error_key: true
  fields:
    service: "service1-audit"
    department: "departmentname"
    project: "service1"
    audit: true

- type: log
  enabled: true
  paths:
    - N:/logs/service1/logstash/logstas*.log
  json:
    keys_under_root: true
    add_error_key: true
  fields:
    service: "service1"
    department: "departmentname"
    project: "service1"

setup:
  template:
    enabled: true
    name: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
    pattern: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
  ilm.enabled: false

output.elasticsearch:
  hosts: ["https://host1:9200","https://host2:9201","https://host3:9202"]
  ssl.certificate_authorities: ['C:\filebeat\certs\ca.crt']
  protocol: https
  username: "filebeat"
  password: "strong-password"
  data_stream: true
  indices:
    - index: "secure-%{[fields.department]:unknown}-%{[fields.service]:unknown}-audit"
      when.equals:
        fields.audit: "true"
    - index: "comp-%{[fields.department]:unknown}-%{[fields.service]:unknown}"
      not:
        when.equals:
          fields.audit: "true"
  bulk_max_size: ${buffer_size}
  workers: 2
  compression_level: 3

The logs are being collected from the paths specified correctly but although the field.audit is populated the logs are still being written to the datastream called: comp-departmentname-service1.

I suspect that this is happening as although the destination indices are specified with proper conditions, the template is always refering to the service which does not have the audit field.

Can you please let me know how I can tackle such issue please or if I am on the wrong track here?

Thanks
Luke