How tell metricbeat to create custom indices?

Elastic Stack Beats
1

Hi there,
how can I tell metricbeat to direct the ingested data into a custom index? What I tried so far did not work.
cheers
Ste

2

Hey @smm,

Why are you trying to store the data in custom indexes?
Included defaults use to be good for most of the cases, and configuring custom indexes uses to be error-prone, as many things need to be taken into account.

A common issue when trying to configure custom indexes is to overlook ILM configuration, take a look to ILM settings, they are used by default since 7.x, and they influence how indexes are created: https://www.elastic.co/guide/en/beats/metricbeat/7.9/ilm.html

3

Can you share the config you used and the problems you ran into?

4

Hi JSoriano,
my customers would like to have installed specific defaults dashboards. But since it is a big, shared central elasticsearch cluster for multiple customers, we use a tailored naming convention for the index names.
Thanks for the hint regarding ILM. Since we only have a golden license we deactivated ILM in the cluster. Nevertheless I will take a look to the link you provided.
Thank you and kind regards
Stefano

5

Not sure I understand why you'd do that?

However if you have a subscription, the you should definitely be chatting to your Support Engineer about this.

6

Hi Warkolm,
so far I had no luck with this tests. I tried:
a) adding these lines at the end of the metricbeat.yml:
setup.template.name: "metricbeat"
setup.template.pattern: "fra-beat-"
--> retstarted metricbeat & elasticsearch
and then did:
/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E setup.kibana.space.id=metrics-docker -E setup.dashboards.index="fra-beat-" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86_64/kibana/choosen/

b) uncommented these again and tried this cli command:
-> deleted the custom space where I was testing and then:
/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E setup.kibana.space.id=metrics-docker -E setup.dashboards.index="fra-beat-ASTERISK" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86_64/kibana/choosen/ --index-management -E setup.template.name="metricbeat" -E setup.template.pattern="fra-beat-*"
having added the bold part.

In a) and b) the data still gets collected in the standard index.
So the point is: how can I tell metricbeat to use a non standard index.
GET /_cat/templates just shows me no new template being generated:
heartbeat-7.8.1 [heartbeat-7.8.1-] 1
metricbeat-7.9.1 [metricbeat-7.9.1-] 1
ilm-history [ilm-history-2*] 2147483647 2
auditbeat-7.8.1 [auditbeat-7.8.1-] 1
filebeat-7.8.1 [filebeat-7.8.1-] 1
metricbeat-7.8.1 [metricbeat-7.8.1-] 1
packetbeat-7.8.1 [packetbeat-7.8.1-] 1
and GET /_template/metricbeat-7.8.1 shows me still the standard values:
{
"metricbeat-7.8.1" : {
"order" : 1,
"index_patterns" : [
"metricbeat-7.8.1-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "metricbeat",
"rollover_alias" : "metricbeat-7.8.1"
},
"mapping" : {
"total_fields" : {
"limit" : "10000"
}
},
"refresh_interval" : "5s",
"query" : {
"default_field" : [
"message",
"tags",
"agent.ephemeral_id",
"agent.id",
...
So it still seems I am doing something wrong.
Perhaps you / someone has an example how this is done properly.
kind regards
Stefano

7

Hi,

This is my filebeat config. I have two indices. one for cisco logs (cisco-beat) and one for other logs (default filebeat index).
Take a look at lines 16, 19-23, and 32-33.
If you don't disable ILM like line 16, your custom indices will be ignored:

Custom index settings are ignored when ILM is enabled.

8

Hello borna_talebi,

thank you very much for your config file!!
And for the hint with ILM!
I will test it today and share my findings with you all.
best regards
Stefano

9

ops - sorry. I mixed up machine learning and ILM.
We deactivated ML. Now in my test environment I will deactivate also ILM for metricbeat. Thank you for pointing me to this and sorry for the confusion.
cheers
Stefano

10

You should still chat to your Support Engineer, they can provide guidance here.

11

Hi there,
it just worked! Let me make some more tests, collect the results and then I will publish it.
cheers
Stefano

1 Like
12

My solution so far:
I had to put this into the metricbeat.yml:
output.elasticsearch:
enabled: true
hosts: ["127.0.0.1:9200"]
index: "fra-beat-*%{[agent.version]}-%{+yyyy.MM.dd}"
...
setup.ilm.enabled: false
setup.template.name: "metricbeat"
setup.template.pattern: "fra-beat-*"

and do this:
/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml setup -E setup.kibana.host=127.0.0.1:5601 -E setup.kibana.username=elastic -E setup.kibana.password=password -E setup.kibana.space.id=metrics-docker -E setup.dashboards.index="fra-beat-*" -E setup.dashboards.directory=/tmp/metricbeat-7.8.1-linux-x86_64/kibana/choosen/

I did this in a separate all-in-one node cluster - 7.8.1 version. Normally a customer would have to change the yml of each and every of his beats in order to add the bold lines.
cheers
Stefano

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.