Set "number_of_replicas" to 0 permanently

Elastic Stack Elasticsearch
1

Hi,

Because of my current Elasticsearch cluster deployment (only one Hot node, not best practice, I know), I need to set the "number_of_replicas" of all my already an newly created indexes to 0.

As I use a Fleet Server with many Agent Policies and integrations, I used to paste in Stack Management > Index Management > Templates > logs (Managed) > Index settings :

image
image1715×533 52.6 KB

As this Index Template is Managed, it seems that for whatever reason (maybe integration updates?), this setting is reset and doesn't persist.

So the current workaround I found is to go in Management > Dev Tools > Console and manually send this PUT request :

PUT /_all/_settings
{
  "index" : {
      "number_of_replicas" : 0
  }
}

But how to set this permanently for all already and newly created indexes please?

Thanks a lot for your help.
Regards.

2

Unfortunately customization like this on Elastic Agent is still pretty bad and you will need to create a custom template for each dataset in each one of your integrations.

Go into Stack Management > Index Management > Component templates and search for custom, you will see multiple @custom templates like these ones:

Screenshot from 2024-04-04 08-57-06

You will need to edit each one of them and add the number_of_replicas settings.

1 Like
3

Thanks a lot @leandrojmp for pointing me to the right direction!
As I have approximately 120 custom component templates, I managed to edit them using a simple bash script :

# Retrieve the list of all component templates
curl --cacert <http_ca.crt> -u <username>:'<password>' -X GET https://<es_master_node>:9200/_component_template | tee elastic_component_template_list.json

# Extract all component templates with pattern name @custom
cat elastic_component_template_list.json | jq . | grep custom | sed 's/^[[:space:]]*"name": "\(.*\)",$/\1/' | tee elastic_component_template_custom_list.txt

# Do some verification / manual edit if needed on elastic_component_template_custom_list.txt

# Update all @custom component templates settings
for i in $(cat elastic_component_template_custom_list.txt); do curl --cacert <http_ca.crt> -u <username>:'<password>' -X PUT "https://<es_master_node>:9200/_component_template/$i" -H 'Content-Type: application/json' -d'{"template": {"settings": {"index": {"number_of_replicas": "0"}}}}'; done

So now, do you thing that everything is fine? No other manual operation needed on my side like mapping custom component templates to index templates?
It seems that it is already set :

image
image1717×721 95.2 KB

Have a great day!
Regards.

4

Did you edit the custom templates or just replaced them?

I'm not sure you can just replace them because they have some metadata that I'm also not sure if this can have any impact or not.

But besides that, you would just need your indices to rollover and get the new template or force a rollover if you want.

I would test this on a couple of indices before applying to all.

5

Actually, you do not .... :slight_smile:

You just need to create the custom component template

logs@custom

And that will apply to all logs-* data streams :slight_smile:

Screenshot 2024-04-04 at 3.55.10 PM
Screenshot 2024-04-04 at 3.55.10 PM1546×994 61.6 KB

Turns out @custom templates are available at

the <type> level

logs@custom

as well as the

<type>-<datastream> level

1 Like
6

Is this new? I knew that something like that was implemented for ingest pipelines, but didn't knew that it was already working for templates as well.

I'm tracking this issue about it: [Fleet] Add support for customizing integration data streams at more levels of granularity · Issue #149484 · elastic/kibana · GitHub and it is still open.

EDIT:

It seems that it was merged on 8.9 according to this PR: [Logs+] Add support for `logs@custom` component template by eyalkoren · Pull Request #95481 · elastic/elasticsearch · GitHub

1 Like
7

A curious thing is that I'm on 8.12.1 and the logs@custom does not exists and it seems that the logs template was never updated to include ignore_missing_component_templates": ["logs@custom"].

So you got this error if you try to preview the logs template.

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "index_template [simulate_template_yka2wonftvyovf3wx0jz2w] invalid, cause [index template [simulate_template_yka2wonftvyovf3wx0jz2w] specifies component templates [logs@custom] that do not exist]",
  "attributes": {
    "error": {
      "root_cause": [
        {
          "type": "invalid_index_template_exception",
          "reason": "index_template [simulate_template_yka2wonftvyovf3wx0jz2w] invalid, cause [index template [simulate_template_yka2wonftvyovf3wx0jz2w] specifies component templates [logs@custom] that do not exist]"
        }
      ],
      "type": "invalid_index_template_exception",
      "reason": "index_template [simulate_template_yka2wonftvyovf3wx0jz2w] invalid, cause [index template [simulate_template_yka2wonftvyovf3wx0jz2w] specifies component templates [logs@custom] that do not exist]"
    }
  }
}

Manually creating an empty logs@custom component template solves this.

1 Like