Maximum limit for indexes

Damian_Brhel · February 28, 2012, 10:48am

Hi,

I have a project which provides to brands monitoring their brand
names. I want to store data separately into different index for each
brand. Is there a limit for number of indexes with the Elastic Search
can real work? Will ES still work if i have 10k - 100k indexes? Each
index will have around 100k new rows every month.

I want to store data separately, beacuse we always select data only
for one brand and I want to use advantage of _open/_close index.

Thank you!

Damian Brhel

kimchy · February 29, 2012, 1:53pm

See this thread: Redirecting to Google Groups, but in general, its pretty expensive to have many indices (even with one shard).

On Tuesday, February 28, 2012 at 12:48 PM, Damian Brhel wrote:

Hi,

I have a project which provides to brands monitoring their brand
names. I want to store data separately into different index for each
brand. Is there a limit for number of indexes with the Elastic Search
can real work? Will ES still work if i have 10k - 100k indexes? Each
index will have around 100k new rows every month.

I want to store data separately, beacuse we always select data only
for one brand and I want to use advantage of _open/_close index.

Thank you!

Damian Brhel

James_Bardin · July 17, 2012, 9:37pm

Hi Shay,

On Wednesday, February 29, 2012 8:53:16 AM UTC-5, kimchy wrote:

See this thread:
Redirecting to Google Groups,
but in general, its pretty expensive to have many indices (even with one
shard).

Could you elaborate a bit on the overhead of indices? A single
index-per-object is gives us a perfect setup for our dataflow, but how far
can this scale reasonably? Though we're starting out at ~20,000 indices,
I'd like to be able to plan for 1 million if possible.

If this isn't too far fetched, which is the more important scaling factor,
node size or node count?

Thanks,
-jim

Clinton_Gormley · July 18, 2012, 8:57am

Hi Jim

Could you elaborate a bit on the overhead of indices? A single
index-per-object is gives us a perfect setup for our dataflow, but how
far can this scale reasonably? Though we're starting out at ~20,000
indices, I'd like to be able to plan for 1 million if possible.

If this isn't too far fetched, which is the more important scaling
factor, node size or node count?

That is indeed far fetched Unless you plan on having hundreds of
thousands of servers...

However, filtered aliases might be what you are after.

I suggest taking a look at Shay's presentation at berlin buzzwords this
year:

clint

Thanks,
-jim

James_Bardin · July 18, 2012, 1:44pm

On Wed, Jul 18, 2012 at 4:57 AM, Clinton Gormley clint@traveljury.com wrote:

That is indeed far fetched Unless you plan on having hundreds of
thousands of servers...

However, filtered aliases might be what you are after.

I suggest taking a look at Shay's presentation at berlin buzzwords this
year:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Now that's the exact information I was looking for (specifically the
"user" dataflow section)!

Thanks!
-jim

Brett_Paden · April 25, 2013, 7:51pm

Elasticsearch Platform — Find real-time answers at scale | Elastic

404s -- is there an update location for this video?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · April 25, 2013, 9:48pm

Just checked the video section under Resources on the new home page, and
found it here

Jörg

Am 25.04.13 21:51, schrieb Brett Paden:

>
http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html
<http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html>

404s -- is there an update location for this video?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Brett_Paden · April 25, 2013, 11:39pm

Danke!

On Thu, Apr 25, 2013 at 5:48 PM, Jörg Prante joergprante@gmail.com wrote:

Just checked the video section under Resources on the new home page, and
found it here

Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/videos/big-data-search-and-analytics/

Jörg

Am 25.04.13 21:51, schrieb Brett Paden:
>
http://www.elasticsearch.org/**videos/2012/06/05/big-data-**
search-and-analytics.htmlhttp://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html
<http://www.elasticsearch.org/**videos/2012/06/05/big-data-**
search-and-analytics.htmlhttp://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

404s -- is there an update location for this video?
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**kiFI0QoZ3v4/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/kiFI0QoZ3v4/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.com elasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.