First of all, greetings everyone,
I am working for a company that actually use Sphinx, but I have the job to
migrate from Sphinx to Elastic Search since it can't handle well the amount
of data that we have.
We have actually about 3 billions documents and growing fast, and a almost
unlimited budget for an Elastic Search cluster.
We are aggregating many data from multiple source, and it's mainly accessed
by user. Some of the data is private and some is public (twitter, etc.) so
we had the idea to split the data into 2 indices (one public and one
private). Basically the idea behind that is that the private index will
have a large amount of shards to do overallocation and routing by users id
since it is always accessed this way. For the public data, we have the idea
of putting it into a big public index, but this for this part, the queries
might be more painful since there is no routing.
This is actually my problem... we ideally want to have response time under
200ms and about 1000 QPS. We will have to ping the 2 indices at almost each
So the killing question.. I was thinking about 50 nodes (32gb RAM with 8
cores) so 50 shards for the private index and 25 for the public maybe... do
you think our goals can be achieved with that amount of nodes and data ?
I have runned some tests with 6 nodes with 1 big index and 1/10 of our data
with mitiged results (I mean there more optimization job to do...) Some of
the responses were very fast and some were still at 1 sec (for the users
with many relations with other users).
Any suggestions might be welcomed at this point...