Oh man, after a few days of tinkering (I say a few days, I've been working
on this for a while..) I'm finally indexing in bulk at a reasonable speed
~2500 docs per second.
I'm going to tweak some settings and see how fast I can get it and then
I'll post my final settings. I think the key was scaling out rather than
scaling up.
Cheers,
James
On Fri, Mar 1, 2013 at 11:18 AM, Jörg Prante joergprante@gmail.com wrote:
It's a matter of distribution of the data. Look where your primay shards
are, since all indexing goes through the primary shards, and how much
resources are there (CPU cores, Memory). Check how your routing distributes
over the shards, it depends on the routing parameter, and if the data
volume of a user varies much in respect to the average user data volume. If
overall data distribution is well, there is no "bottleneck".Increasing shards has an upper limit. You can increase the number of
shards as long as your machines can handle the distributed indexing load.
If the limit is exceeded, just add machines, it's as easy as that.Jörg
Am 01.03.13 11:42, schrieb james.lewis@7digital.com:
This is interesting - we use routing to make sure that all of a users
documents will be indexed in the same shard. So we're not just hitting the
bulk api with documents, we're also supplying a route for each document
based on the user id. Anyone know how this might have an impact on the
performance of a batch load?--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**PQizsDan8Zc/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/PQizsDan8Zc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited. 7digital Limited. Registered office: Unit F, Lower Ground
Floor, 5-25 Scrutton Street, Zetland House London EC2A 4HJ. Registered in
England and Wales. Registered No. 04843573.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.