On Thu, Oct 9, 2014 at 6:34 PM, Kevin Burton burtonator@gmail.com wrote:
On Wednesday, October 8, 2014 12:07:30 AM UTC-7, Jörg Prante wrote:
With ES, you can go up to the bandwidth limit the OS allows for writing
I/O (if you disable throttling etc.)
This means, if you write to one shard, it can be as fast as writing to
thousands of shards in parallel in summary. There is an OS limit for file
system buffers so the more shards, the more RAM is recommended.
If OS restricts file descriptor limits and you want to write to shards,
you can estimate you need a peak of 100-200 file descriptors for active
merges etc. (this varies from version to version and from setting to
setting). ES does not impose a limit here.
These resource demands and custom configurations are not related to the
choice SSD or HDD.
There's no way that's true... for example, if you are on HDD, and you're
index is not in memory, AND you're serving queries (which is a realistic
use case) then there is NO way you can write at the full IO of the disk.
It's just physically impossible.
If ES has been able to solve that problem then they could win a nobel
prize :-p
SSD is 2-3 orders of magnitude faster than HDD ... so yes, it is related
to the choice of SSD or HDD.
... it's entirely possible I'm misinterpreting what you're saying though.
Yeah. You could disable throttling and write faster. But that'd be
stupid if you need query performance to stay constant.
You asked about the load on my shards. Its not at all evenly distributed.
Update and query rate varies a ton depending on the index.
The are per-index overheads just for having that many indexes but from my
perspective they are pretty small. Some stuff isn't right, for example
Elasticsearch distributes the memory used to buffer writes evenly across
indexes that have received writes in the past few minutes.
Actually serving searches across all the shards is another story though.
For the most part we lay things out so that each search request hits 1/3 -
2 of our servers. We're probably more concerned about that then most
people, though, because we frequently perform actions that have high-ish
per shard overhead like the phrase suggester. Oversubscribing is ok for
indexes that don't get much query traffic but otherwise we make sure
everything is spread out as evenly as possible. We do that by cranking up
the *index *allocation factor and by adding total_shards_per_node = 1 to
our highest traffic indexes. Cranking up the index allocation spreads the
shards of each index out pretty evenly but sometimes Elasticsearch will
smash them together during "exciting" situations like when a node drops
out. total_shards_per_node is a hard limit so even when things get
exciting it'll keep those shards away from each other.
If your wondering about write speed its only kinda related to shards. The
two things you can do to speed up writing is to make sure that all shards
you are writing to are evenly spread out across your nodes and to crank up
the number of segments that are allowed to exist. More segments lower the
write amplification factor both for IO and CPU usage. BUT they cost more
to search. If you plan to do a ton of writes at once then never again you
can crank up the segments during the write and then run an optimize to
squash the segments together. That works great if you never update any of
the documents in the index. Its even ok to index more stuff into the
index. It starts to be annoying if you do lots of updates (I do) because
you get into situations where your big segments have lots of deleted
document in them. Then you have to merge them and that has lots of
overhead. If they get big enough the merge policy will sometimes refuse to
pick them for merging too! Its kind of a degenerate case though.
Looping back around to shards for write performance - the number of
segments allowed is a per shard thing. Its set on the whole index but it
applies per shard. Say the max segments per size tier is 10 and the number
of shards is 5. Then you get 50 segments per tier. So if you double the
number of shards you double the number of segments per tier so writes are
faster. The trouble is there is no command to squash shards together. So
maybe the better solution is to double the number of segments per tier and
then run optimize, because squashing shards together is optimize's job.
OTOH see the caveats above to optimize.
Nik
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2kEhe%2B-YabODeFA-6cLY9BZrn7WC0K7CVuo689RoHMAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
