On Fri, Aug 29, 2014 at 1:27 PM, Rob Blackin email@example.com wrote:
We are trying to implement a 5 TB, 10 Billion item Elasticsearch cluster.
The key is an integer and the item data is fairly small.
We're running around 5.5TB right now without a problem. The biggest
annoyance is that rolling restarts take time proportional to how much data
We have much larger documents then you have so we only store 181 million or
so. Our documents are interactively maintained - a consistent portion of
them are updated daily with some creates and a few rare deletes.
You might want to think about how you do sharding - look into routing to
see if you can get away with oversubscribing on shards. You might also
look into using multiple indexes as well. Shay gave a talk on how you
could subdivide one large set of data into multiple indexes to help
things. One 5TB index would be difficult to maintain. As are any shards
that are more then, say, 20GB. Just shuffling those shards from system to
system for rebalancing gets expensive. Merges on those shards have a
higher upper bound on disk io and cache thrash.
We seem to run into issues around loading. Seems to slow down as the index
Check on your merge rate. This is old but it'll give you some idea of what
is going on:
You can tune this a bit - especially if your data comes in spurts.
We are doing this on EC2 i2.xlarge nodes.
How many documents/TB do you think we can load per node max?
So if we can do 2 Billion each then we need 5 nodes. We are trying to size
I can't talk to Amazon because we use physical machines. We use 18
machines with two reasonably nice Intel ssds per machine, 96GB of ram, and
pretty sizeable CPUs and it isn't really enough to handle the query load we
want to throw at it. I imagine the shape of your load is going to be of a
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1-59_4WQyKGFOsWBDmZd8iYu9agQPszwh80rB8g8vQ4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.