Has anyone come up with a maintainable way to set index.number_of_replicas cluster wide?

Elastic Stack Elasticsearch
1

Thanks to budget issues, I'm trying to slim down the footprint of my Elastic stack.

While not ideal, running in single-node mode would make things easier to manage.

But, as far as I can tell, Elastic Agent creates everything to have at least 1 replica. Due to the defaults I'm sure.

That results in any single node cluster always using twice as much space as it should, and always being "yellow". Not exactly the setup I need.

index.number_of_replicas is the setting I think I need to update. Setting it to 0 should do the trick. But it is an index level setting, not cluster (though I think it might have once been a cluster wide setting?). Which means I need to either update the settings on every index after it is created, or somehow add the setting to the index template.

Manually updating either index or index template settings is not going to work. Elastic Agent creates hundreds of both. And I'm pretty sure any changes to managed index templates will get removed when a new version comes along.

Which brings me to the question in the topic title. Has anyone come up with a maintainable way to set index.number_of_replicas cluster wide?

Is there a way to tell Elastic Agent to set it when it creates a new index? Or to add the setting to the index templates it creates?

Thanks in advance!

2

Well actually no... there may be a replica specified but it will not be allocated so twice the storage is not actually utilized on a single node cluster, the cluster will just remain yellow.

There is an intention to this... it is a way to indicate to the user they are at risk of data loss.
That they need to intentionally change settings as part of acknowledging that risk.

No, I do not believe so... perhaps someone else will have a better idea.

There is a concept of priorities with templates but I do not think it was intended for this.

You can certainly manually change the replicas to 0 in the template, but yes they will get re-written when you update integration.

A yellow cluster is fine, as long as you understand why.

You could also daily / hourly etc run a script or when ever detected manually set the replicas to 0.

Perhaps someone else will have a better idea.

3

Ooh... I did not know that. Thanks. :slight_smile:

I get that, but I'd also like warnings to go away when I've acknowledged them. As it is, I have to keep "acknowledging" the warning every single time a new index is created.

Sure, but what about other conditions that cause a yellow cluster? Those won't throw up a warning if I just ignore the "yellow" status.

I was kinda thinking that might be what I'd have to do...

Thanks for the quick reply!

4

A yellow cluster is caused by unallocated replica shards, I don't think that there is any other condition that would cause a yellow cluster.

Unfortunately this is true, any customization on Elastic Agent, be it templates, mappings or ingest pipelines, adds a lot of work that makes it pratically impossible to do.

The easiest solution in this case would really be a script running on some interval to set the number of replicas to zero.

5

So, one solution is if you don't proxy stuff through Logstash, you can use the setup.template.settings.index configuration in your beats config to set the number_of_replicas value.

'Course that only works with beats directly, not sure how you'd do it with agent.

6

Maybe it's not exactly what you ask for but what we do with our index (we also run primary only due to budget constraints) is that we regularly do something like:

PUT index*/_settings?expand_wildcards=all
{
  "index.number_of_replicas": 0
}

where index* is then of course matching the index namnes, for example .ds-logs* or .metrics* or whatever it can be. We run it manually since we want to have some "pre-flight" checks but I suppose a simple solution could be to run something like that from a crontab or similar.

7

Hi

I had a similar issue in our cluster. I see 2 ways to disable the replica's:

  • After creation of the index, you disable them
  • Before creation of the index, you disable them (index template, component template which is added to every index template, ...)

As per the previous comment, you can disable replica's on created indicies:
Please note that the I had to add the first command to also alter the system indices.

PUT /.ds-*/_settings
{
  "index":{
    "number_of_replicas": 0
  }
}

PUT /_settings
{
  "index":{
    "number_of_replicas": 0
  }
}

Disabling replica's on future indices, can be done via index/component templates:
E.g.

PUT _component_template/metrics-apm.service_summary.10m@custom
{"template":{"settings":{"index":{"lifecycle":{"name":"Justx-Metrics-Default"},"number_of_replicas":"0"}}},"_meta":{"package":{"name":"apm"},"managed_by":"fleet","managed":true}}

PUT _component_template/metrics-system.network@custom
{"template":{"settings":{"index":{"lifecycle":{"name":"Justx-Metrics-Default"},"number_of_replicas":"0"}}},"_meta":{"package":{"name":"system"},"managed_by":"fleet","managed":true}}

PUT _component_template/metrics-elastic_agent.auditbeat@custom
{"template":{"settings":{"index":{"lifecycle":{"name":"Justx-Metrics-Default"},"number_of_replicas":"0"}}},"_meta":{"package":{"name":"elastic_agent"},"managed_by":"fleet","managed":true}}

I had to alter a lot of templates, so I'll share some of the commands used to gather the names for the custom components (Fleet/Elastic Agent). With some smart regexes you could easily get the above commands to run them all at once:
TIP: they slightly differ between apm, system and elastic-agent

GET /_component_template/metrics-*custom
GET /_component_template/metrics-apm.*custom
GET /_component_template/metrics-system.*custom
GET /_component_template/metrics-elastic_agent.*custom

GET /_component_template/logs-*custom?filter_path=component_templates.name
GET /_component_template/logs-system.*custom
GET /_component_template/logs-elastic_agent.*custom
GET /_component_template/logs-apm.*custom

PS: I can share all the commands if you would like to :slight_smile:

Coming back to the index templates, I prefer to have some component templates which are used over multiple index templates, e.g.:
TIP: Keep in mind that if you configure something via Kibana, the last screen has a tab "request" where you can get below requests.

PUT _component_template/logs-justx-settings
{
  "template": {
    "settings": {
      "index": {
        "number_of_replicas": "0",
        "default_pipeline": "calculate_lag"
      }
    }
  }
}

And using them to construct the template (with ILM):

PUT _ilm/policy/logs-trace-prd
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "5gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "readonly": {},
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          },
          "migrate": {
            "enabled": false
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

PUT _index_template/logs-trace-prd
{
  "version": 1,
  "priority": 500,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs-trace-prd"
        }
      }
    }
  },
  "index_patterns": [
    "logs-trace-prd"
  ],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "composed_of": [
    "logs-mappings",
    "data-streams-mappings",
    "logs-justx-settings",
    "logs-justx-mappings"
  ]
}

Conflicting settings will be overwritten by the last component defining them:

  "composed_of": [
    "logs-mappings",
    "data-streams-mappings",
    "logs-justx-settings",
    "logs-justx-mappings"
  ]

Good luck!

8

I've only run this in dev so far, and I do need to get it running on cron, but it might help someone else down the road.

Basically I curl the json of all the templates I want to change, then send it through jq to make the changes I need, and back to curl to upload the new templates. I made sure to set the priority to one higher than what the ES docs says they use.

#!/usr/bin/env bash
TEMPLATES=$(curl -s -X GET "https://$ES_BACKEND_USER:$ES_BACKEND_PASS@$ES_HOST:9200/_index_template/metrics*");
if [[ $TEMPLATES == '{"index_templates":[]}' ]]; then
   echo "Metrics Template Adjustment - Templates do not exist yet.";
   sleep 10;
   continue
else
  TEMPLATES_EMPTY="present"
  echo "Metrics Template Adjustment - Templates found and loaded."
  curl -s -X PUT -H "Content-Type: application/json" \
       -d '{"template": {"settings": { "index": { "number_of_replicas": "0" }}}}' \
       "https://$ES_BACKEND_USER:$ES_BACKEND_PASS@$ES_HOST:9200/_component_template/projectname-metrics-adjustment"
  echo $TEMPLATES | jq -c '.index_templates[].name |= "projectname-" + . | .index_templates[].index_template.priority=500 | .index_templates[].index_template.composed_of += ["projectname-metrics-adjustment"] | .index_templates[] ' \
        | while IFS= read -r obj; do
            NAME=$(echo $obj | jq -r .name); echo "Metrics Template Adjustment - uploading $NAME."; NO_NAME=$(echo $obj | jq 'del(.name)'); TEMPLATE=$(echo $NO_NAME | jq '.index_template'); curl -X PUT -H "Content-Type: application/json" -d"$TEMPLATE" "https://$ES_BACKEND_USER:$ES_BACKEND_PASS@$ES_HOST:9200/_index_template/$NAME"
          done
fi

While indices created before the script ran don't have the settings, all the ones created after do.

2 Likes
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.