# Filebeat manage multiple write indices

**URL:** <https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 21, 2020, 3:07am UTC](https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760 "2020-10-21T03:07:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![keshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keshara/32/63979_2.png) [@keshara](https://discuss.elastic.co/u/keshara)\
**Post date:** [October 21, 2020, 3:07am UTC](https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760/1 "2020-10-21T03:07:48Z")

</div>

According to the link =\> [https://www.elastic.co/guide/en/beats/filebeat/6.8/ilm.html#\_advanced\_ilm\_settings](https://www.elastic.co/guide/en/beats/filebeat/6.8/ilm.html#_advanced_ilm_settings)  
I am trying to create multiple indices from filebeat output to elasticsearch.

Here is my filebeat.yml.

```auto
    - type: log
      enabled: true
      paths:
       - C:\es790\filebeat-7.9.0\jsonlog1.json
      tags: ["xxxx"]
      processors:
      - add_fields:
          target: ''
          fields:
            diff: adminportal

    - input_type: log
      paths:
       - C:\es790\filebeat-7.9.0\jsonlog1.json
      tags: ["yyyy"]
      processors:
      - add_fields:
          target: ''
          fields:
            diff: ruleengine

    setup.template.enabled: false
      
    output.elasticsearch:
      index: logs-%{event.diff}
      hosts: ["localhost:9200"]
      pipeline: "pipeline_test"
      ilm.enabled: true

```

When I start the filebeat, I get following error

> ERROR instance/beat.go:951 Exiting: error initializing publisher: unsupported format expression "event.diff" in index  
> Exiting: error initializing publisher: unsupported format expression "event.diff" in index

Looks like the line  
`output.elasticsearch.index: logs-%{event.diff}`  
causing the error. But the link I attached above mention such possibility.

Any help.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 21, 2020, 3:35pm UTC](https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760/2 "2020-10-21T15:35:27Z")

</div>

It seems our documentation is incorrect. I have tried your configuration and the one in our docs but got the same error.

The solution is to use `logs-%{[event.diff]}`.

---

<div class="post-metadata">

**Author:** ![keshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keshara/32/63979_2.png) [@keshara](https://discuss.elastic.co/u/keshara)\
**Post date:** [October 22, 2020, 2:24am UTC](https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760/3 "2020-10-22T02:24:44Z")

</div>

Thanks for getting back.

Yes, change the index name as `logs-%{[event.diff]}` helped to avoid the error.

But that given details in the documentation does not help to manage multiple indices from Filebeat when ILM part of the config.

According to the doc, it says..

> If you change the index name, you must also set the template name, template pattern, rollover alias, and lifecycle name. The best way to set these is through an Elasticsearch template. It’s possible to disable the template loading in Filebeat and specify these settings in your own template.

I did the exactly what it says, meaning creating TEMPLATE in elasticsearch and apply index-pattern/ lifecycle-name/ lifecycle-rollover alias. But that didn't help creating multiple indices & rollover when needed.

After doing many test, I found out following config works

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - C:\es790\filebeat-7.9.0\sample.log
  processors:
  - add_fields:
      target: ''
      fields:
        diff: type1_log

- type: log
  enabled: true
  paths:
   - C:\es790\filebeat-7.9.0\sample1.log
  processors:
  - add_fields:
      target: ''
      fields:
        diff: type2_log

setup.template.enabled: false
setup.ilm.enabled: false
setup.ilm.pattern: "{now/d}-000001"
  
output.elasticsearch:
  index: "xxx-%{[diff]}"
  hosts: ["localhost:9200"]

```

Additionally, I had to create Index alias for both index as well as their respective Elasticsearch-Templates

Index Alias =\>

```auto
PUT xxx-type1_log-000001
{
  "aliases": {
    "xxx-type1_log": {
      "is_write_index": true
    }
  }
}

```

Index Template =\>

```auto
{
  "index_patterns": [
    "xxx-type1_log-*"
  ],
  "order": 98,
  "settings": {
    "number_of_shards": 10,
    "number_of_replicas": 1,
    "index.lifecycle.name": "policy_name",
    "index.lifecycle.rollover_alias": "xxx-type1_log"
  },
  "mappings": {},
  "aliases": {}
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 19, 2020, 4:25am UTC](https://discuss.elastic.co/t/filebeat-manage-multiple-write-indices/252760/4 "2020-11-19T04:25:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.