Is it safe to set action.auto_create_index to false?

Richard_Zhang · January 5, 2025, 2:20pm

Hello everyone,

We're using Elastic Cloud v8.11.1 and facing a problem where the automatically created index does not have the desired field mapping, so we're considering disable auto create index.

while this document says we can just achieve this with a PUT request
index-creation, I also find a few posts in this community saying disable auto index creation will cause problems, for example:

impact-on-kibana
security related issue
failed to start after disabling index auto creation

So can anyone help clarify whether it is safe to entirely disable index auto creation by setting action.auto_create_index to false?

If it could cause problems to other components like Kibana, security etc, then I believe it is necessary to add a warning in the official document about set it to false.

RainTown · January 5, 2025, 7:12pm

Can I gently suggest you should prioritise the (likely eminently solvable and IMO more important) problem with the mapping.

Tell us more about that please,

Which specific fields get mapping wrong, what do you expect/want, what have you done already? Do you have an index template. If not, why not?

Richard_Zhang · January 6, 2025, 2:49pm

Thank you RainTown for the quick response.

Yes, using index template to ensure correct field mappings is one of the options for us. but the field mappings for our index is quite complex --- Normally, we build the field mapping based on a series of configuration files in our system. this should happen before any document indexing happens. In normal case, it would be working well.

but in some cases, the index creation fails somehow. after that, when documents are indexed, the index gets created automatically with default field mapping. That's the problem.

I agree on that we could try to enhance our process to avoid such cases but I'm eager to know if there is suggestion about the auto_create_index setting.

Is it safe to set it to false for a standard elastic stack deployment? Is this a dangerous action? If there are already known issues is being caused by this setting. I believe it's worth mentioning it in official document.

RainTown · January 6, 2025, 9:46pm

FWUW I still think you are looking at the wrong problem.

Index Templates have priorities. Probably it’s just a few fields that are problematic, right? (If not, you really are in trouble)

So create an effectively failsafe, simple, low priority index template that has correct mapping, limited to just your troublesome fields. When you get the wider solution fully implemented, great, but in meantime you’d have IMO a better workaround than what you are exploring.

Richard_Zhang · January 7, 2025, 5:27am

Thanks. I believe Index template is a good option in many cases.

We haven't used index template for this part of our system, and there are quite a lot of fields are defined programmingly based on a few of configuration files. So it does not sound easy for us to switch to index template.

btw, although templates have priorities, a index can only use a single template, right? so a low priority template might not work. the document says

If a new data stream or index matches more than one index template, the index template with the highest priority is used.

(appreciate it and It's pleasure to discuss this topic with you.)

RainTown · January 8, 2025, 4:23pm

No, an index can take settings from multiple templates. If templates conflict , the higher priority template settings wins on that specific setting.

That’s the essence of my suggestion.

Fix the problematic fields with a simple low priority template.
Work on the wider template generating issue, eventually generating a higher priority, more complete template.

Richard_Zhang · January 9, 2025, 2:59pm

Thank you RainTown.

an index can take settings from multiple templates. If templates conflict , the higher priority template settings wins on that specific setting.

The official doc says only one template will be used. Is this something that I'm somehow misunderstanding? or maybe this doc is misleading and need correction?

If a new data stream or index matches more than one index template, the index template with the highest priority is used.

RainTown · January 9, 2025, 9:44pm

AFAIK What I wrote applies. Test it and you’ll see. I’m on holiday right now so can’t validate myself.

The doc you quote can arguably be more clearly worded, maybe that snippet is just clarifying that higher (priority) numbers wins is in case of conflicting settings.

Richard_Zhang · January 10, 2025, 1:55pm

Thank you. Let's try it out and let you know.

Richard_Zhang · January 11, 2025, 3:05pm

Hi @RainTown

As per my testing, the template with higher priority just blindly overwrites the one with lower priority. pls see the details as below.

# low-priority template with two settings and two field mappings.
PUT /_index_template/template_rich1
{
  "index_patterns": ["rich*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "refresh_interval": "10s"
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date"
        }
      }
    },
    "aliases": {
      "mydata": { }
    }
  },
  "priority": 100,
  "version": 3,
  "_meta": {
    "description": "my custom"
  }
}

# high-priority template with 1 different setting and 1 different field mapping.
PUT /_index_template/template_rich2
{
  "index_patterns": ["rich*"],
  "template": {
    "settings": {
      "number_of_shards": 2
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "updated_at": {
          "type": "date"
        }
      }
    }
  },
  "priority": 200

}


PUT /rich0/

GET /rich0/

The created index:

{
  "rich0": {
    "aliases": {},
    "mappings": {
      "properties": {
        "updated_at": {
          "type": "date"
        }
      }
    },
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "2",
        "provided_name": "rich0",
        "creation_date": "1736607359424",
        "number_of_replicas": "1",
        "uuid": "o1duNn0nS3mEhDBNmpanIA",
        "version": {
          "created": "8500003"
        }
      }
    }
  }
}

As it shows, the new index rich0 only reflects the mappings and settings from the high-priority template.

The above index templates API is the new version instead of the legacy one. So maybe you were talking about the behavior of the legacy one?

Legacy Index template

Composable Index template

RainTown · January 12, 2025, 12:37am

Thanks for checking, seems indeed I am simply mistaken and spreading old/fake/outdated news.

If so, as seems likely, please accept my sincere apologies.

Plenty tines people here insist (rightly) to only trust the vendor documentation, and discourage third party information / “documentation”. I seem to have fallen into that trap myself. I can quote several links , even recent ones, that say what I said, but I won’t. Cos it looks like it’s plain wrong.

I’ll look again when back from my vacation.

RainTown · January 12, 2025, 12:49am

I guess this is borderline official:

From:

When both legacy and composable templates exist and they match with the same index pattern, the legacy template will be ignored.

If two composable templates point to the same index pattern, the template with the highest priority will be used.

If two legacy templates point to the same index pattern, the templates are merged, with higher-order templates overriding lower-order ones. If the order is the same, the templates are sorted by name and merged accordingly.

This seems clear.

But in that very same document:

In this example, you have two (composable) templates—ct1 and ct2—both targeting the same index pattern test_index-. However, ct2 has a higher priority (1) than ct1 (0). Therefore, when you create an index that matches the pattern test_index-, the settings and mappings defined in ct2 will be applied before ct1. If there are the same settings in the ct1 and ct2 templates, the ct2 template will overwrite.

Note use of “before ct1” there, which I think is slightly ambiguous as ct1 should be ignored.

Richard_Zhang · January 14, 2025, 2:20am

Yes, I agree with you, ct1 is just ignored and ct2 is applied with all of its settings/mappings/aliases.

Legacy index template has the order-and-merge behavior, which is useful to avoiding wrong field mappings(the problem for this topic). but as its name indicates, I feel like it's a recommended option for us since it's a legacy feature and might be discontinued someday.

So we're circling back, setting auto_create_index as a whitelist is a valid options to solve the problem we're facing.

Actually there is already another topic going on What is .ent-search-*-logs-*,-.ent-search-*? where is it from?

I've been so happy discussing this topic with you by which we've made a good understanding about index templates. You're so helpful.

Topic		Replies	Views
PUT index working despite action.auto_create_index=false? Elasticsearch	4	6091	July 15, 2019
Unexpected behaviour for auto_create_index Elasticsearch docker	5	928	January 18, 2022
Allow_auto_create set to false in all templates after uprgade to 8.12.0 Elasticsearch	1	364	May 16, 2024
Disable automatic index creation in elasticsearch Elasticsearch	3	11804	February 4, 2021
Elasticsearch's action.auto_create_index setting impact on Kibana Kibana	3	2919	February 28, 2018

Is it safe to set action.auto_create_index to false?

Related topics