Hi everyone, we're considering moving from Sphinx to Elastic Search, but I
want to make sure it is a good fit before rewriting our infrastructure.
Currently we have 20 dual octo core 2690 machines with 32GB of ram. We
handle about 2,000 queries per second with the existing setup, but we have
some pain points that I believe ES can help with.
Index rotation. Our index is about 30GB in size, but the nature of
Sphinx means that each week when our dataset is updated, we have to reindex
the entire dataset. Further, we have to do this on all 20 machines. We
currently reindex on one central machine, then rsync the data to the 20
main servers, then perform rolling restarts. Rsyncing 30GB of data to 20
machines takes too long, and it is only going to get worse as the number of
MySQL... Since Sphinx uses MySQL as a datastore, we are going to reach a
point where our database becomes a bottleneck. Our MySQL servers have no
problem handling the current load of 5,000 or so queries per second, but DB
servers are expensive to scale, and I would rather store the data in ES and
skip SQL completely.
My understanding is that ES shards the index across machines as they are
added. We have worked with cassandra in the past. The concept seems very
I know it's difficult to predict, but how is query performance with ES? Is
CPU or memory/IO the main bottleneck? We're moving to a new datacenter
where we will have 30 or so dual hex core machines but with 72GB of memory
each instead of the 32 we have in the current machines. Since ES shards the
index across machines, we should have no problem storing everything in
memory, so i'm guessing we would still be cpu bound (as we are with sphinx.
It isn't a problem, just wondering if I can expect the same).
Has anyone here moved a large scale sphinx cluster over to ES? Any gotcha's
that i'm overlooking? Any super easy migration plan you found that will cut
our development time from months down to days/weeks lol? I know, wishful
Thanks in advance!
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5934aa1c-84ee-4177-ad05-f449d4fe14f8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.