Is there limitation how many indices could I create in ES cluster? and performance?

Elastic Stack Elasticsearch
1

Hi, all

Is there limitation how many indices could I create in ES cluster? and Does
the number of indices affect performance?
I have used DATE as indice for logs from MMO game servers. That give me
chance to remove old data.

But right now, I'm considering use userid as indice, that means there might
be milliion users, as well as million indices created.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2f0d724-f1b1-40b4-8457-318a195ebdd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Each index comes with a cost and probably having million of indices will require a lot of machines.
Also the cluster state will be a way too big so it could affect cluster stability.

You will probably have at the end of the day a lot of small indices.

I mean: don't do this! :slight_smile:

Share indices between users but use routing to make sure all docs for a given user go to the same shard.

Having rolling indices is a good idea though.

Here are my first thoughts about this.
Hope this helps.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 mars 2015 à 02:46, zehong yin timesking@gmail.com a écrit :

Hi, all

Is there limitation how many indices could I create in ES cluster? and Does the number of indices affect performance?
I have used DATE as indice for logs from MMO game servers. That give me chance to remove old data.

But right now, I'm considering use userid as indice, that means there might be milliion users, as well as million indices created.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2f0d724-f1b1-40b4-8457-318a195ebdd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BEB80606-4151-4308-B32C-556906E302E3%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

You may use a single index with enough shards for users and use routing for
accessing the shard where a user ID has the docs indexed. See also shard
overallocation

and
https://groups.google.com/forum/#!msg/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ

Jörg

On Sat, Mar 14, 2015 at 10:46 AM, zehong yin timesking@gmail.com wrote:

Hi, all

Is there limitation how many indices could I create in ES cluster? and
Does the number of indices affect performance?
I have used DATE as indice for logs from MMO game servers. That give me
chance to remove old data.

But right now, I'm considering use userid as indice, that means there
might be milliion users, as well as million indices created.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b2f0d724-f1b1-40b4-8457-318a195ebdd7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b2f0d724-f1b1-40b4-8457-318a195ebdd7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGN5ugQWN%3DW_Fs19_tsiGxw4uhwE_3Ye30_eCGFWJ%2BY9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.