I understand the reason for the throttling, especially on shared
environments with slow I/O such as EC2, I just can't believe how low the
default is. 
I was hoping to start a conversation of everyone's settings
and how they are tuned.
In our setup, shard recovery should theoretically occur only after a full
reindex where we index to zero replicas and then increase the replica
count. Therefore all segments should be merged by then and we should never
be in a situation where shards are recovering and segments are merging. Our
initial settings for recovery and merge throttling are both 60mb. Of
course, shard recovery can occur after a node drops out, in which case
throttling is a good thing since other larger issues might be causing
problems.
Cheers,
Ivan
On Wed, Jul 3, 2013 at 12:22 AM, Alexander Reelsen alr@spinscale.de wrote:
Hi Ivan,
just to make sure everyone understands, why elasticsearch opted for rather
low defaults. First not everyone is using SSDs (even though you should be).
In case you are not, low settings as defaults ensure, that even under high
load (recovery + merges + searches, so the worst case from a load point of
view) elasticsearch stays as responsive as possible, at the cost of being
under a higher load for longer time.
And yes, tuning these values is highly recommended (elasticsearch has to
offload this decision to the administrator, as he knows the underlying
hard, especially if it is virtualized).
--Alex
On Tue, Jul 2, 2013 at 11:59 PM, Ivan Brusic ivan@brusic.com wrote:
Did some monitoring of the transport size with the merge throttling set
to an arbitrarily high number. Transaction sizes easily topped 100mb, so
the default value of 20mb is very low for our setup. Just an FYI for those
that are used to pre-0.90 behaviors. Will
tune indices.store.throttle.max_bytes_per_sec next.
Cheers,
Ivan
On Mon, Jul 1, 2013 at 10:06 AM, Ivan Brusic ivan@brusic.com wrote:
One of the issues that I noticed with my recent upgrade to 0.90 is the
slowness of the shard recovery process. After some digging, I found out
that the recovery process is now throttled whereas previously it was
unlimited.
The defaults for both merge and recovery throttling is 20mb. I wonder
what settings everyone else is running. Upping the recovering throttling to
80mb works well with no consequences during our limited tests. Have not
looked into merge throttling yet.
We are running local virtual machines (no EC2/Rackspace/etc...) with
platter-based storage.
Ivan
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.