# Metricbeat forcing to use manually created template

**URL:** <https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [July 5, 2024, 10:27pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634 "2024-07-05T22:27:59Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 5, 2024, 10:28pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/1 "2024-07-05T22:28:00Z")

</div>

Hi Folks, I have an APIKey in my Elasticsearch which allows only index creation with specific patterns such as "app-abc-\*". It does not allow index template creations. The company has a default-template which is applied to all indexes.

Well, If I start my metricbeat with name "app-abc" on index configuration I get an error because I need to set "setup.template" parameters.

So, I requested admin to create an empty template with name which metricbeat needs, and it worked, but since the company has a default-template the default one is used, and that's ok.

But my only concern is : I need to create a empty template just to bypass metricbeat error, why ? Is there a better way to do that ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 5, 2024, 11:48pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/2 "2024-07-05T23:48:28Z")

</div>

If you have a template for your indices did you simply try to set the following in metricbeat.yml

`setup.template.enabled: false`

> **[Configure Elasticsearch index template loading | Metricbeat Reference \[8.14\]...](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-template.html)**

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 7:41pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/3 "2024-07-06T19:41:02Z")

</div>

It does not works, I got error:

`setup.template.name and setup.template.pattern have to be set if index name is modified`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 8:00pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/4 "2024-07-06T20:00:18Z")

</div>

Hi @Ronaldo_Lanhellas Please share your metricbeat.yml and what version you are on.

By the way, setting up an index per app is an anti-pattern, as it potentially creates many small indices, which is not efficient. Curious why you want to do this... not to say there is not a reason but curious why.

Putting many apps in the same index / data stream is the best practice and then filter on the Query Side

for 8.14 this is a complete working metricbeat.yml

That will write to custom alias, index or data stream

```auto
metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: my-metrics

```

```auto
GET _cat/indices/my-metrics?v

# Result
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open my-metrics tYuUH8ovRJC0rSNj2nbECw 1 1 291 0 566.4kb 566.4kb 566.4kb

```

You can also review [this post](https://discuss.elastic.co/t/metricbeat-and-heartbeat-8-x-custom-index-name-should-pickup-from-template-but-creates-always-bedefault-index-name/337557/9)

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 8:19pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/5 "2024-07-06T20:19:30Z")

</div>

Using metricbeat 8.6.1, here is my yml file

```auto
metricbeat.config.modules:
  path: ./modules.d/*.yml

processors:
  - add_fields:
      target: org
      fields:
        cluster: kconnect-oracle-blu

cloud.id: xx

output:
  elasticsearch: 
   allow_older_versions: true
   api_key: xx
   index: kconnect

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0640

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: kconnect
setup.template.pattern: kconnect

```

About setting up an index per app, actually, I'm setting up an index per context in my org. So a lot of kafka connect brokers will write to the same datastream.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 8:24pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/6 "2024-07-06T20:24:01Z")

</div>

> [@Ronaldo\_Lanhellas](#):
>
> About setting up an index per app, actually, I'm setting up an index per context in my org. So a lot of kafka connect brokers will write to the same datastream.

Ok that makes sense....

Now precise question

A) did you already manually create an index template, ILM policy etc... to write to the data stream?

B) You want metricbeat to setup (which mean Create for you) the index template etc.

A or B ... which one? (because you config above does neither correctly)

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 8:27pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/7 "2024-07-06T20:27:51Z")

</div>

Option A, the SRE team created everything for us and just need to use. The only thing we have permission to create is the indexes with pattern "kconnect-\*"

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 8:31pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/8 "2024-07-06T20:31:04Z")

</div>

When you want to use templates / ILM that are manually created then you must set both

```auto
setup.template.enabled: false
setup.ilm.enabled: false

```

So then I would use....

```auto
metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: konnect-%{[agent.version]}

```

or index name as you choose etc...

` index: konnect-prod`

Note you said SRE said

`kconnect-*` so there must be a `-` and something after it

I updated the Topic Title to perhaps help someone else since this is not about creating a template it is about using a template

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 8:44pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/9 "2024-07-06T20:44:27Z")

</div>

If I change the index value to something different from default one, metricbeat throws error if I don't set template name and pattern.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 8:56pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/10 "2024-07-06T20:56:41Z")

</div>

Did you try to run the one I provided... exactly as I provided......

I will try to go back to 8.6.1

I just ran exactly this on 8.6.2 and it worked fine

```auto
metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

cloud.id: sladkjfhaskldfjh

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]
  api_key: salkjdfhasldkfjh
  index: konnect-%{[agent.version]}

```

There i some disconnect... are you sure there is already a template..

Can you please try exactly what I gave you no more ... no less...

Just add your `cloud.id` and `api_key`

Please show the exact / all the errors...

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 9:04pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/11 "2024-07-06T21:04:18Z")

</div>

I see

` allow_older_versions: true`

How old of a version of elasticsearch are you trying to connect to? &7.X?

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 9:27pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/12 "2024-07-06T21:27:50Z")

</div>

Worked now @stephenb , thx for your help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 9:29pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/13 "2024-07-06T21:29:39Z")

</div>

What was the fix so others know?

---

<div class="post-metadata">

**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\
**Post date:** [July 6, 2024, 9:44pm UTC](https://discuss.elastic.co/t/metricbeat-forcing-to-use-manually-created-template/362634/14 "2024-07-06T21:44:34Z")

</div>

For some reason I just set the `setup.template.enabled: false` and worked.