# Filebeat cannot create custom index - even when ilm is disabled

**URL:** <https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 28, 2022, 1:40pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688 "2022-10-28T13:40:55Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 28, 2022, 1:40pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/1 "2022-10-28T13:40:55Z")

</div>

Hi Team,  
I would like to ask for help as I cannot create custom index even when setup.ilm.enabled: is set to false.

**Below is my filebeat config. Thanks.**

setup.ilm.enabled: false  
setup.template.enabled: true  
setup.template.name: "testing-%{[agent.version]}"  
setup.template.pattern: "testing-\*"  
setup.template.fields: "fields.yml"  
setup.template.overwrite: false  
setup.template.settings:  
index.number\_of\_shards: 1

output.elasticsearch:  
hosts: ["172.30.169.50:9200"]  
ssl.verification\_mode: "none"  
username: "elastic"  
password: "m4gN3s1@.123"  
indices:

- index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"  
when.equals:  
fields:  
event.module: "cisco"
- index: "testing-%{[agent.version]}-fortinet-%{+yyyy.MM.dd}"  
when.equals:  
fields:  
event.module: "fortinet"

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 29, 2022, 12:05am UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/2 "2022-10-29T00:05:44Z")

</div>

Hi @gisellecarballo

When you use a filebeat modules. The index name is actually defined in the module. (Well technically the input settings). And input settings override output settings. That's why you're not seeing your output setting take effect.  
So you have to overwrite it in the module.

Yes that should be documented better. It is in a docs but not very clear... See [here](https://www.elastic.co/guide/en/beats/filebeat/current/advanced-settings.html)

It looks like you're using the fortinet in Cisco modules.

So perhaps go into the module ymls

And set the index there example

```auto
- module: cisco
  asa:
    enabled: true
    var.paths: ["/var/log/cisco-asa.log"]
    var.input: "file"
    index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"

```

Then you don't even need to set the index setting in the elasticsearch output.

Think that'll work for you. Let us know

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 31, 2022, 2:20pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/3 "2022-10-31T14:20:21Z")

</div>

Hi @stephenb ,

Unfortunately, your suggestion does not work with my case. I added both the index on the cisco and fortinet module but no luck.  
Is there any work around you know ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/4 "2022-10-31T15:03:48Z")

</div>

> [@gisellecarballo](#):
>
> Unfortunately, your suggestion does not work with my case. I added both the index on the cisco and fortinet module but no luck.  
> Is there any work around you know ?

Huh ... I will take a look later today..

When you say it does not work ... what is the result?

What is the name of the index that is being written too?

What version of Filebeat and Elastic.

Can you share your cicso.yml

The more you provide the more we can help.

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 31, 2022, 3:36pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/5 "2022-10-31T15:36:13Z")

</div>

Hi @stephenb ,  
I really appreciate you getting back to me.  
Filebeat, kibana and elasticsearch versions are all 7.17.5  
When I tried the below configurations, log does not even come thru on the default index, in short no syslogs seen at all.

```auto

###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

###filebeat/modules.d/fortinet.yml####
- module: fortinet
  firewall:
    enabled: true
    var.input: udp
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5009
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

###filebeat/filebeat.yml####
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log

- type: syslog
  enabled: false
  paths:
    - /var/log/*.log

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1@.123"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 3:54pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/6 "2022-10-31T15:54:13Z")

</div>

> [@gisellecarballo](#):
>
> When I tried the below configurations, log does not even come thru on the default index, in short no syslogs seen at all.

Well that is telling...

I would take out ALL the custom index stuff and first see if you are getting logs to the default filebeat-7.17.5 indices...

Then Report Back..'

I always tell people get the defaults working first .. then customize

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 31, 2022, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/7 "2022-10-31T16:10:01Z")

</div>

Hi @stephenb ,

Able to get the default working.  
Please see image below.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66b2756dcfacb5f6031475de77d4bbcaabd1905f.jpeg)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 4:12pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/8 "2022-10-31T16:12:20Z")

</div>

Ok I will look at it a bit later... it is something simple that we are missing.  
You can take out the fields

`setup.template.fields: "fields.yml"`  
unless you modified it.. (which I hope you did not)

Also when you run the above config... did you look at the filebeat logs... it should tell you why it can not index...

Also assuming this is cut-npaste

Should be also are you trying to do http or https? it is unclear to me?

```auto
output.elasticsearch:
  hosts: ["172.30.169.50:9200"]
  ssl.verification_mode: "none"
  username: "elastic"
  password: "m4gN3s1@.123"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/9 "2022-10-31T16:29:57Z")

</div>

@gisellecarballo

Ohh Darn ... I forgot the `input.` prefix

```auto
###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

```

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 31, 2022, 4:43pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/10 "2022-10-31T16:43:20Z")

</div>

I did not modify fields.yml  
Below is the filebeat logs which I think would be relevant on our issue.  
Elasticsearch is only http.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/b/abb804e4c092b5f94e457b16c9d98750c105a4fe.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/11 "2022-10-31T17:02:41Z")

</div>

Apologies.. Really hard to read / work with screenshots... text is much better in the future.

Well you have other issues... your cluster / number of shards is full

I suggestion you clean up your cluster / address this first...

 ![Screen Shot 2022-10-31 at 10.00.34 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fb0e07d97a0ad8325ca805e815cfe25712701ea5.png)

That is probably why it can not create the new indices...

So you need to get your cluster healthy / cleaned up first or increase

> **[Cluster-level shard allocation and routing settings | Elasticsearch Guide...](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-limit)**

If you increase the shard limit that is just a work around not a real fix... it will cause problems at some point.

Also I always ask to show the command you ran...

it looks like you ran the following for lots of details

`filebeat -e -d "*" `

you can just run

`filebeat -e`

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [October 31, 2022, 7:08pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/12 "2022-10-31T19:08:30Z")

</div>

I have cleaned up my cluster and yet custom indices were not working. I wonder what im missing here.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 8:05pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/13 "2022-10-31T20:05:23Z")

</div>

Well now you need to run filebeat without the -d "\*" and look at the logs... the logs are very descriptive...

Did you fix the

` input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2022, 10:04pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/14 "2022-10-31T22:04:37Z")

</div>

@gisellecarballo Ok I found some issues and have it figured out...

Here are 2 fully functional approaches

I think this will work for you ..... using very close to your original...  
This is a fully functional filebeat.yml... I just put the module directly in the filebeat.yml to test.  
You were very close in the beginning then I gave you bad advice... see the 2nd example.

```auto
filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

```

or define the index int in the module BUT the event.dataset is not available must be defined manually, this is where I gave you bad advice and it was not working.

```auto
filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"
      input.index: "testing-%{[agent.version]}-cisco.asa-%{+yyyy.MM.dd}"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true

output.elasticsearch:
  hosts: ["localhost:9200"]

```

Both of these produce...

```auto
GET _cat/indices/?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open testing-7.17.3-cisco.asa-2022.10.31 iz5_7S5YQziApDl2TML1gQ 1 1 88 0 104kb 104kb

```

---

<div class="post-metadata">

**Author:** ![gisellecarballo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gisellecarballo/32/112755_2.png) [@gisellecarballo](https://discuss.elastic.co/u/gisellecarballo)\
**Post date:** [November 2, 2022, 12:46pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/15 "2022-11-02T12:46:17Z")

</div>

Hi @stephenb ,

In my case, the second one works.  
Putting index on each filebeat module.  
Thank you for your effort on this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2022, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-cannot-create-custom-index-even-when-ilm-is-disabled/317688/16 "2022-11-30T14:46:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.