Filebeat manage multiple write indices

[keshara (Keshara Dorakumbura)]

According to the link => https://www.elastic.co/guide/en/beats/filebeat/6.8/ilm.html#_advanced_ilm_settings
I am trying to create multiple indices from filebeat output to elasticsearch.

Here is my filebeat.yml.

    - type: log
      enabled: true
      paths:
       - C:\es790\filebeat-7.9.0\jsonlog1.json
      tags: ["xxxx"]
      processors:
      - add_fields:
          target: ''
          fields:
            diff: adminportal

    - input_type: log
      paths:
       - C:\es790\filebeat-7.9.0\jsonlog1.json
      tags: ["yyyy"]
      processors:
      - add_fields:
          target: ''
          fields:
            diff: ruleengine

    setup.template.enabled: false
      
    output.elasticsearch:
      index: logs-%{event.diff}
      hosts: ["localhost:9200"]
      pipeline: "pipeline_test"
      ilm.enabled: true

When I start the filebeat, I get following error

ERROR instance/beat.go:951 Exiting: error initializing publisher: unsupported format expression "event.diff" in index
Exiting: error initializing publisher: unsupported format expression "event.diff" in index

Looks like the line
output.elasticsearch.index: logs-%{event.diff}
causing the error. But the link I attached above mention such possibility.

Any help.

[kvch (Noémi Ványi)]

It seems our documentation is incorrect. I have tried your configuration and the one in our docs but got the same error.

The solution is to use logs-%{[event.diff]}.

[keshara (Keshara Dorakumbura)]

Thanks for getting back.

Yes, change the index name as logs-%{[event.diff]} helped to avoid the error.

But that given details in the documentation does not help to manage multiple indices from Filebeat when ILM part of the config.

According to the doc, it says..

If you change the index name, you must also set the template name, template pattern, rollover alias, and lifecycle name. The best way to set these is through an Elasticsearch template. It’s possible to disable the template loading in Filebeat and specify these settings in your own template.

I did the exactly what it says, meaning creating TEMPLATE in elasticsearch and apply index-pattern/ lifecycle-name/ lifecycle-rollover alias. But that didn't help creating multiple indices & rollover when needed.

After doing many test, I found out following config works

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - C:\es790\filebeat-7.9.0\sample.log
  processors:
  - add_fields:
      target: ''
      fields:
        diff: type1_log

- type: log
  enabled: true
  paths:
   - C:\es790\filebeat-7.9.0\sample1.log
  processors:
  - add_fields:
      target: ''
      fields:
        diff: type2_log

setup.template.enabled: false
setup.ilm.enabled: false
setup.ilm.pattern: "{now/d}-000001"
  
output.elasticsearch:
  index: "xxx-%{[diff]}"
  hosts: ["localhost:9200"]

Additionally, I had to create Index alias for both index as well as their respective Elasticsearch-Templates

Index Alias =>

PUT xxx-type1_log-000001
{
  "aliases": {
    "xxx-type1_log": {
      "is_write_index": true
    }
  }
}

Index Template =>

{
  "index_patterns": [
    "xxx-type1_log-*"
  ],
  "order": 98,
  "settings": {
    "number_of_shards": 10,
    "number_of_replicas": 1,
    "index.lifecycle.name": "policy_name",
    "index.lifecycle.rollover_alias": "xxx-type1_log"
  },
  "mappings": {},
  "aliases": {}
}