Metricbeat forcing to use manually created template

Elastic Stack Elasticsearch
1

Hi Folks, I have an APIKey in my Elasticsearch which allows only index creation with specific patterns such as "app-abc-*". It does not allow index template creations. The company has a default-template which is applied to all indexes.

Well, If I start my metricbeat with name "app-abc" on index configuration I get an error because I need to set "setup.template" parameters.

So, I requested admin to create an empty template with name which metricbeat needs, and it worked, but since the company has a default-template the default one is used, and that's ok.

But my only concern is : I need to create a empty template just to bypass metricbeat error, why ? Is there a better way to do that ?

2

If you have a template for your indices did you simply try to set the following in metricbeat.yml

setup.template.enabled: false

3

It does not works, I got error:

setup.template.name and setup.template.pattern have to be set if index name is modified

4

Hi @Ronaldo_Lanhellas Please share your metricbeat.yml and what version you are on.

By the way, setting up an index per app is an anti-pattern, as it potentially creates many small indices, which is not efficient. Curious why you want to do this... not to say there is not a reason but curious why.

Putting many apps in the same index / data stream is the best practice and then filter on the Query Side

for 8.14 this is a complete working metricbeat.yml

That will write to custom alias, index or data stream

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: my-metrics

GET _cat/indices/my-metrics?v

# Result
health status index      uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   my-metrics tYuUH8ovRJC0rSNj2nbECw   1   1        291            0    566.4kb        566.4kb      566.4kb

You can also review this post

5

Using metricbeat 8.6.1, here is my yml file

metricbeat.config.modules:
  path: ./modules.d/*.yml

processors:
  - add_fields:
      target: org
      fields:
        cluster: kconnect-oracle-blu

cloud.id: xx

output:
  elasticsearch: 
   allow_older_versions: true
   api_key: xx
   index: kconnect

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0640

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: kconnect
setup.template.pattern: kconnect

About setting up an index per app, actually, I'm setting up an index per context in my org. So a lot of kafka connect brokers will write to the same datastream.

6

Ok that makes sense....

Now precise question

A) did you already manually create an index template, ILM policy etc... to write to the data stream?

B) You want metricbeat to setup (which mean Create for you) the index template etc.

A or B ... which one? (because you config above does neither correctly)

7

Option A, the SRE team created everything for us and just need to use. The only thing we have permission to create is the indexes with pattern "kconnect-*"

8

When you want to use templates / ILM that are manually created then you must set both

setup.template.enabled: false
setup.ilm.enabled: false

So then I would use....

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: konnect-%{[agent.version]}

or index name as you choose etc...

index: konnect-prod

Note you said SRE said

kconnect-* so there must be a - and something after it

I updated the Topic Title to perhaps help someone else since this is not about creating a template it is about using a template

9

If I change the index value to something different from default one, metricbeat throws error if I don't set template name and pattern.

10

Did you try to run the one I provided... exactly as I provided......

I will try to go back to 8.6.1

I just ran exactly this on 8.6.2 and it worked fine

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.enabled: false
setup.ilm.enabled: false

setup.kibana:

cloud.id: sladkjfhaskldfjh

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]
  api_key: salkjdfhasldkfjh
  index: konnect-%{[agent.version]}

There i some disconnect... are you sure there is already a template..

Can you please try exactly what I gave you no more ... no less...

Just add your cloud.id and api_key

Please show the exact / all the errors...

11

I see

allow_older_versions: true

How old of a version of elasticsearch are you trying to connect to? &7.X?

12

Worked now @stephenb , thx for your help.

1 Like
13

What was the fix so others know?

14

For some reason I just set the setup.template.enabled: false and worked.

1 Like