A cautionary tale about index autocreation

I'm going to write this now before I forget about it.

We're going to turn off automatic index creation. We find it hides
deployment mistakes. Everything will look like it is working but you're
Italian text is suddenly missing English stop words. Correcting the
problem basically requires blowing away and recreating the index any way.

If you can use the feature either because the defaults are fine or you can
use templates or whatever, then more power to you. It really is super
convenient. I think, at least for us, turning it off makes more sense.
Maybe you too should think about it.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Index autocreation is definitely one of the features you want to turn off
in production. It's super convenient, but can lead to all sorts of problems
if you're not careful. We also disable a few other potentially dangerous
actions in production. This is snippet from our configuration:

Require explicit index creation

action.auto_create_index: false

Protect against accidental close/delete operations

on all indices. You can still close/delete individual

indices

action.disable_close_all_indices: true
action.disable_delete_all_indices: true

Disable ability to shutdown nodes via REST API

action.disable_shutdown: true

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, October 24, 2013 6:48:01 PM UTC-7, Dan Everton wrote:

Index autocreation is definitely one of the features you want to turn off
in production. It's super convenient, but can lead to all sorts of problems
if you're not careful.

Hear, hear.

We've been running these stricter configurations in production for
Bonsai.io for ages for exactly those reasons. The difference is a periodic
source of confusion for some of our customers, and probably our largest
single source of Elasticsearch-related support questions at bonsai.io.

While dynamic index creation is a pretty "cool" feature, I'd love to see it
changed to an opt-in setting, and have proposed as much in the "Breaking
changes for 1.0" issue on
github: https://github.com/elasticsearch/elasticsearch/issues/4007

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86d8f9e6-d7ac-4345-be01-88df3f3878bf%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Index autocreation is definitely one of the features you want to turn off
in production.

This all depends who "you" are. For instance, for logging users, index
auto-creation is exactly what you want. So the "correct" setting depends
entirely on how you use Elasticsearch.

A strong theme in Elasticsearch is "just get out of the way and make it
easy to get going" and I think auto index-creation fits nicely with that
approach. Should you later want to turn it off, then fine, it's easy to do.

Something similar going for the _all field or dynamic mapping etc. They
all make it easy to get started, but can be configured to be stricter
should you desire it.

Possibly what we need instead is a page in the docs explaining these
settings so that users don't need to discover them only after being bitten

clint

On 2 December 2013 16:48, Nick Zadrozny nick@onemorecloud.com wrote:

On Thursday, October 24, 2013 6:48:01 PM UTC-7, Dan Everton wrote:

Index autocreation is definitely one of the features you want to turn off
in production. It's super convenient, but can lead to all sorts of problems
if you're not careful.

Hear, hear.

We've been running these stricter configurations in production for
Bonsai.io for ages for exactly those reasons. The difference is a periodic
source of confusion for some of our customers, and probably our largest
single source of Elasticsearch-related support questions at bonsai.io.

While dynamic index creation is a pretty "cool" feature, I'd love to see
it changed to an opt-in setting, and have proposed as much in the "Breaking
changes for 1.0" issue on github:
https://github.com/elasticsearch/elasticsearch/issues/4007

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86d8f9e6-d7ac-4345-be01-88df3f3878bf%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKS-P0_%3DhOJJHt8DoKLt2SBdwXYh6tW%3DN4q8N2hh-ZFz7w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Mon, Dec 2, 2013 at 11:12 AM, Clinton Gormley clint@traveljury.comwrote:

Index autocreation is definitely one of the features you want to turn
off in production.

This all depends who "you" are. For instance, for logging users, index
auto-creation is exactly what you want. So the "correct" setting depends
entirely on how you use Elasticsearch.

A strong theme in Elasticsearch is "just get out of the way and make it
easy to get going" and I think auto index-creation fits nicely with that
approach. Should you later want to turn it off, then fine, it's easy to do.

I like this argument even though I'm the one that originally complained
about it. I can see wanting to use it.

Something similar going for the _all field or dynamic mapping etc. They

all make it easy to get started, but can be configured to be stricter
should you desire it.

Maybe ironically my use case calls for turning all those off as well.

Possibly what we need instead is a page in the docs explaining these
settings so that users don't need to discover them only after being bitten

+1. A page with links to all the gotcha settings and a discussion on why
you might want to change the defaults would be perfect. As of now there
are blogs to read but none of them are really comprehensive because they
typically about that user's user case.

I think it is nice to keep in mind that it is normal for a service's
defaults not to be correct for production/high load use. Elasticsearch
kind of spoils us because the defaults actually are right in so many cases.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1RxvFWDePKWh9f5ChGiV1upjn3XOgdu9xktN1GCKj6Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

+1 for Clint

I like the principle "convenience over configuration".

Auto index creation may hurt in a few cases, but it is easy to turn it off.

In my production setup, auto doc ID creation sucks. So I can turn it off if
I want to. But I'd never blame ES for having auto doc ID creation turned on
by default.

An overview of Elasticsearch's convenience philosophy and why it is better
instead of hiding features would help though.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnqYCuGarrze2OVKmiYxDtfP6e140k3O8Ch6scf57JpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

+1 from me, too :slight_smile:

Maybe an option is to say somewhere (docs? blog? comments in config file?
all of the above?) that ES is developer-friendly and some settings should
be checked before going to production.

If you need any help, I'd be glad to contribute.

Speaking of contribuitions, here's a crazy idea: how about a GitHub repo?
It would be easier for the community to add their experience and even have
branches for use-cases like logging.
On Dec 2, 2013 6:51 PM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

+1 for Clint

I like the principle "convenience over configuration".

Auto index creation may hurt in a few cases, but it is easy to turn it off.

In my production setup, auto doc ID creation sucks. So I can turn it off
if I want to. But I'd never blame ES for having auto doc ID creation turned
on by default.

An overview of Elasticsearch's convenience philosophy and why it is better
instead of hiding features would help though.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnqYCuGarrze2OVKmiYxDtfP6e140k3O8Ch6scf57JpQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2dYjZrMVngC_3b%2BYfez2UoekRhJD4Jx_tYn4KNfrM%2B3A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Doesn't look like this exists in the official docs yet, but this blog has a
good list of settings to check:

On Monday, December 2, 2013 1:34:33 PM UTC-6, Radu Gheorghe wrote:

+1 from me, too :slight_smile:

Maybe an option is to say somewhere (docs? blog? comments in config file?
all of the above?) that ES is developer-friendly and some settings should
be checked before going to production.

If you need any help, I'd be glad to contribute.

Speaking of contribuitions, here's a crazy idea: how about a GitHub repo?
It would be easier for the community to add their experience and even have
branches for use-cases like logging.
On Dec 2, 2013 6:51 PM, "joerg...@gmail.com <javascript:>" <
joerg...@gmail.com <javascript:>> wrote:

+1 for Clint

I like the principle "convenience over configuration".

Auto index creation may hurt in a few cases, but it is easy to turn it
off.

In my production setup, auto doc ID creation sucks. So I can turn it off
if I want to. But I'd never blame ES for having auto doc ID creation turned
on by default.

An overview of Elasticsearch's convenience philosophy and why it is
better instead of hiding features would help though.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnqYCuGarrze2OVKmiYxDtfP6e140k3O8Ch6scf57JpQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a8ce5fd5-77f9-4d21-a82d-4830f7a8a7f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The problem with "action.auto_create_index: false" is that it disables
index templates... So it is basically useless if you are using one

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0a69a36-6497-4a51-a144-48f31a6b2312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.