# How tell metricbeat to create custom indices? **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [September 29, 2020, 6:03am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297 "2020-09-29T06:03:20Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 29, 2020, 6:03am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/1 "2020-09-29T06:03:20Z")
Hi there, how can I tell metricbeat to direct the ingested data into a custom index? What I tried so far did not work. cheers Ste ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [September 29, 2020, 4:17pm UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/2 "2020-09-29T16:17:06Z")
Hey @smm, Why are you trying to store the data in custom indexes? Included defaults use to be good for most of the cases, and configuring custom indexes uses to be error-prone, as many things need to be taken into account. A common issue when trying to configure custom indexes is to overlook ILM configuration, take a look to ILM settings, they are used by default since 7.x, and they influence how indexes are created: [https://www.elastic.co/guide/en/beats/metricbeat/7.9/ilm.html](https://www.elastic.co/guide/en/beats/metricbeat/7.9/ilm.html) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 29, 2020, 11:44pm UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/3 "2020-09-29T23:44:08Z")
Can you share the config you used and the problems you ran into? ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 3:35am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/4 "2020-09-30T03:35:43Z")
Hi JSoriano, my customers would like to have installed specific defaults dashboards. But since it is a big, shared central elasticsearch cluster for multiple customers, we use a tailored naming convention for the index names. Thanks for the hint regarding ILM. Since we only have a golden license we deactivated ILM in the cluster. Nevertheless I will take a look to the link you provided. Thank you and kind regards Stefano ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 30, 2020, 3:41am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/5 "2020-09-30T03:41:18Z")
> [@smm](#): > > Since we only have a golden license we deactivated ILM in the cluster. Not sure I understand why you'd do that? However if you have a subscription, the you should definitely be chatting to your Support Engineer about this. ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 3:43am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/6 "2020-09-30T03:43:34Z")
Hi Warkolm, so far I had no luck with this tests. I tried: a) adding these lines at the end of the metricbeat.yml: setup.template.name: "metricbeat" setup.template.pattern: "fra-beat-_" --\> retstarted metricbeat & elasticsearch and then did: /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E setup.kibana.space.id=metrics-docker -E setup.dashboards.index="fra-beat-_" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86\_64/kibana/choosen/ b) uncommented these again and tried this cli command: -\> deleted the custom space where I was testing and then: /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E setup.kibana.space.id=metrics-docker -E setup.dashboards.index="fra-beat-ASTERISK" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86\_64/kibana/choosen/ **--index-management** -E setup.template.name="metricbeat" -E setup.template.pattern="fra-beat-\*" having added the bold part. In a) and b) the data still gets collected in the standard index. So the point is: how can I tell metricbeat to use a non standard index. GET /\_cat/templates just shows me no new template being generated: heartbeat-7.8.1 [heartbeat-7.8.1-_] 1 metricbeat-7.9.1 [metricbeat-7.9.1-_] 1 ilm-history [ilm-history-2\*] 2147483647 2 auditbeat-7.8.1 [auditbeat-7.8.1-_] 1 filebeat-7.8.1 [filebeat-7.8.1-_] 1 metricbeat-7.8.1 [metricbeat-7.8.1-_] 1 packetbeat-7.8.1 [packetbeat-7.8.1-_] 1 and GET /\_template/metricbeat-7.8.1 shows me still the standard values: { "metricbeat-7.8.1" : { "order" : 1, "index\_patterns" : [ "metricbeat-7.8.1-\*" ], "settings" : { "index" : { "lifecycle" : { "name" : "metricbeat", "rollover\_alias" : "metricbeat-7.8.1" }, "mapping" : { "total\_fields" : { "limit" : "10000" } }, "refresh\_interval" : "5s", "query" : { "default\_field" : [ "message", "tags", "agent.ephemeral\_id", "[agent.id](http://agent.id)", ... So it still seems I am doing something wrong. Perhaps you / someone has an example how this is done properly. kind regards Stefano ---
**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\ **Post date:** [September 30, 2020, 5:12am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/7 "2020-09-30T05:12:12Z")
Hi, > [@smm](#): > > Perhaps you / someone has an example how this is done properly. This is my [filebeat config](https://gist.github.com/bornatalebi/05d59c06447bd32118a2beab864befa7#file-filebeat-configs). I have two indices. one for cisco logs (cisco-beat) and one for other logs (default filebeat index). Take a look at lines 16, 19-23, and 32-33. If you don't disable ILM like line 16, your custom indices will be [ignored](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es): > Custom `index` settings are ignored when ILM is enabled. ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 5:28am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/8 "2020-09-30T05:28:30Z")
Hello borna\_talebi, thank you very much for your config file!! And for the hint with ILM! I will test it today and share my findings with you all. best regards Stefano ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 8:52am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/9 "2020-09-30T08:52:09Z")
ops - sorry. I mixed up machine learning and ILM. We deactivated ML. Now in my test environment I will deactivate also ILM for metricbeat. Thank you for pointing me to this and sorry for the confusion. cheers Stefano ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 30, 2020, 8:52am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/10 "2020-09-30T08:52:59Z")
You should still chat to your Support Engineer, they can provide guidance here. ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 11:03am UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/11 "2020-09-30T11:03:21Z")
Hi there, it just worked! Let me make some more tests, collect the results and then I will publish it. cheers Stefano ---
**Author:** ![smm](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@smm](https://discuss.elastic.co/u/smm)\ **Post date:** [September 30, 2020, 12:35pm UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/12 "2020-09-30T12:35:27Z")
My solution so far: I had to put this into the metricbeat.yml: output.elasticsearch: enabled: true hosts: ["127.0.0.1:9200"] _index: "fra-beat-\*%{[agent.version]}-%{+yyyy.MM.dd}"_ ... _ **setup.ilm.enabled: false** _ _**[setup.template.name](http://setup.template.name): "metricbeat"**_ _ **setup.template.pattern: "fra-beat-\*"** _ and do this: /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E [setup.kibana.space.id](http://setup.kibana.space.id)=metrics-docker -E setup.dashboards.index="fra-beat-\*" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86\_64/kibana/choosen/ I did this in a separate all-in-one node cluster - 7.8.1 version. Normally a customer would have to change the yml of each and every of his beats in order to add the bold lines. cheers Stefano ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 28, 2020, 2:35pm UTC](https://discuss.elastic.co/t/how-tell-metricbeat-to-create-custom-indices/250297/13 "2020-10-28T14:35:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.