Maximum limit for indexes

Elastic Stack Elasticsearch
1

Hi,

I have a project which provides to brands monitoring their brand
names. I want to store data separately into different index for each
brand. Is there a limit for number of indexes with the Elastic Search
can real work? Will ES still work if i have 10k - 100k indexes? Each
index will have around 100k new rows every month.

I want to store data separately, beacuse we always select data only
for one brand and I want to use advantage of _open/_close index.

Thank you!

Damian Brhel

2

See this thread: Redirecting to Google Groups, but in general, its pretty expensive to have many indices (even with one shard).

On Tuesday, February 28, 2012 at 12:48 PM, Damian Brhel wrote:

Hi,

I have a project which provides to brands monitoring their brand
names. I want to store data separately into different index for each
brand. Is there a limit for number of indexes with the Elastic Search
can real work? Will ES still work if i have 10k - 100k indexes? Each
index will have around 100k new rows every month.

I want to store data separately, beacuse we always select data only
for one brand and I want to use advantage of _open/_close index.

Thank you!

Damian Brhel

3

Hi Shay,

On Wednesday, February 29, 2012 8:53:16 AM UTC-5, kimchy wrote:

See this thread:
Redirecting to Google Groups,
but in general, its pretty expensive to have many indices (even with one
shard).

Could you elaborate a bit on the overhead of indices? A single
index-per-object is gives us a perfect setup for our dataflow, but how far
can this scale reasonably? Though we're starting out at ~20,000 indices,
I'd like to be able to plan for 1 million if possible.

If this isn't too far fetched, which is the more important scaling factor,
node size or node count?

Thanks,
-jim

4

Hi Jim

Could you elaborate a bit on the overhead of indices? A single
index-per-object is gives us a perfect setup for our dataflow, but how
far can this scale reasonably? Though we're starting out at ~20,000
indices, I'd like to be able to plan for 1 million if possible.

If this isn't too far fetched, which is the more important scaling
factor, node size or node count?

That is indeed far fetched :slight_smile: Unless you plan on having hundreds of
thousands of servers...

However, filtered aliases might be what you are after.

I suggest taking a look at Shay's presentation at berlin buzzwords this
year:

clint

Thanks,
-jim

5

On Wed, Jul 18, 2012 at 4:57 AM, Clinton Gormley clint@traveljury.com wrote:

That is indeed far fetched :slight_smile: Unless you plan on having hundreds of
thousands of servers...

However, filtered aliases might be what you are after.

I suggest taking a look at Shay's presentation at berlin buzzwords this
year:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Now that's the exact information I was looking for (specifically the
"user" dataflow section)!

Thanks!
-jim

6

Elasticsearch Platform — Find real-time answers at scale | Elastic

404s -- is there an update location for this video?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Just checked the video section under Resources on the new home page, and
found it here

Jörg

Am 25.04.13 21:51, schrieb Brett Paden:

>
http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html
<http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html>

404s -- is there an update location for this video?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Danke!

On Thu, Apr 25, 2013 at 5:48 PM, Jörg Prante joergprante@gmail.com wrote:

Just checked the video section under Resources on the new home page, and
found it here

Elasticsearch Platform — Find real-time answers at scale | Elastichttp://www.elasticsearch.org/videos/big-data-search-and-analytics/

Jörg

Am 25.04.13 21:51, schrieb Brett Paden:

>
http://www.elasticsearch.org/**videos/2012/06/05/big-data-**

search-and-analytics.htmlhttp://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

<http://www.elasticsearch.org/**videos/2012/06/05/big-data-**

search-and-analytics.htmlhttp://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

404s -- is there an update location for this video?

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**kiFI0QoZ3v4/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/kiFI0QoZ3v4/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.