Tuning for high update and delete rate

Elastic Stack Elasticsearch
1

Howdy,

One of our indices has a fairly high document count (around 100 million)
and a high update and delete rate. In any given day, maybe a quarter of
those documents will be updated, replaced, or deleted. Given this, the
default merge settings don't seem to be keeping up and the percentage of
deleted documents slowly creeps up. A lot of these documents are child
documents so this seems to also increase memory pressure as the parent ID
cache is growing along with the delete count. It only seems to clear when a
whole segment is deleted.

I've looked in to the merge and store settings and I'm wondering what the
best course of action to address this is? So far I've increased the
reclaim_delete_weight from 2.0 to 3.0 and changed max_merge_at_once and
segments_per_tier to 5. Store throttle is still enabled, but is set to
80mb/s and the hardware should be capable of at least that. I'm not
completely sure that's the right direction as they don't really seem to
have made any difference. Increasing the throttle rate doesn't seem to have
reduced the store throttle time metric, and the deleted document ratio is
still increasing.

Anyone else had any experience with this sort of indexing load? In case
I've made a typo somewhere, our actual settings look like this:

indices.store.throttle.max_bytes_per_sec: 80mb
index.merge.policy.reclaim_deletes_weight: 3.0
index.merge.policy.max_merge_at_once: 5
index.merge.policy.segments_per_tier: 5

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

not sure, where your problem now is exactly. Do you have file system
troubles, or is your main problem the memory consumption of the parent
child cache?

First, you should go with the latest elasticsearch version, parent child is
constantly improved. Second you could add more nodes to prevent memory
issues with parent child.

Do you monitor your system and see the parent/child cache growing? If so,
what are the numbers? Also, what is a high update/delete rate in numbers?

The merge policy settings also have a couple of special settings regarding
deletes, maybe you can play around with them, see

--Alex

On Thu, Oct 24, 2013 at 12:10 AM, Dan Everton dan@iocaine.org wrote:

Howdy,

One of our indices has a fairly high document count (around 100 million)
and a high update and delete rate. In any given day, maybe a quarter of
those documents will be updated, replaced, or deleted. Given this, the
default merge settings don't seem to be keeping up and the percentage of
deleted documents slowly creeps up. A lot of these documents are child
documents so this seems to also increase memory pressure as the parent ID
cache is growing along with the delete count. It only seems to clear when a
whole segment is deleted.

I've looked in to the merge and store settings and I'm wondering what the
best course of action to address this is? So far I've increased the
reclaim_delete_weight from 2.0 to 3.0 and changed max_merge_at_once and
segments_per_tier to 5. Store throttle is still enabled, but is set to
80mb/s and the hardware should be capable of at least that. I'm not
completely sure that's the right direction as they don't really seem to
have made any difference. Increasing the throttle rate doesn't seem to have
reduced the store throttle time metric, and the deleted document ratio is
still increasing.

Anyone else had any experience with this sort of indexing load? In case
I've made a typo somewhere, our actual settings look like this:

indices.store.throttle.max_bytes_per_sec: 80mb
index.merge.policy.reclaim_deletes_weight: 3.0
index.merge.policy.max_merge_at_once: 5
index.merge.policy.segments_per_tier: 5

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I guess the biggest concern for me is that the deleted document percentage
always creeps up. The secondary effect is that the heap usage is strongly
correlated as well because of the parent ID cache. I was hoping that by
tweaking the merge settings we'd be able to get Elasticsearch to be more
aggressive about expunging deletes and keep the deleted percentage closer
to 10%. Right now we have to periodically do optimize calls just to keep it
below 50%. So I'm hoping the reclaim deletes weight setting should be
enough. We're also going to be testing disabling the store throttle.

We're still on 0.90.2 but have an upgrade to 0.90.5 scheduled soon. Because
of the ID cache stats bug in 0.90.2 we don't really have a good idea of
what's going on with the ID cache.

As for rates, we're averaging about 75 updates per second of which 50 of
those will be delete operations. The update rate is very spiky though (user
driven) so can go as high as 3000 updates per second (so about 2000 deletes
per second). The majority of the document being changed are small child
documents of much larger parent documents. The index has about 25 million
documents.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Dan,

What do your performance metrics say is going on? Is your disk pegged?
CPU? Historical metrics with something like Ganglia are best but you can
get disk utilization with
iostate -dmx 3 100
and get a very good idea of what is going on right now.

Nik

On Sun, Oct 27, 2013 at 6:45 PM, Dan Everton dan@iocaine.org wrote:

I guess the biggest concern for me is that the deleted document percentage
always creeps up. The secondary effect is that the heap usage is strongly
correlated as well because of the parent ID cache. I was hoping that by
tweaking the merge settings we'd be able to get Elasticsearch to be more
aggressive about expunging deletes and keep the deleted percentage closer
to 10%. Right now we have to periodically do optimize calls just to keep it
below 50%. So I'm hoping the reclaim deletes weight setting should be
enough. We're also going to be testing disabling the store throttle.

We're still on 0.90.2 but have an upgrade to 0.90.5 scheduled soon.
Because of the ID cache stats bug in 0.90.2 we don't really have a good
idea of what's going on with the ID cache.

As for rates, we're averaging about 75 updates per second of which 50 of
those will be delete operations. The update rate is very spiky though (user
driven) so can go as high as 3000 updates per second (so about 2000 deletes
per second). The majority of the document being changed are small child
documents of much larger parent documents. The index has about 25 million
documents.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Just to follow this up. We've managed to sort out our settings to cope with
the deletion rate we have. The key seemed to be disabling the store
throttle completely. Elasticsearch is running on physical instances with
fast(ish) disks so the throttle just seemed to be getting in the way. The
final settings were

indices.store.throttle.type: none
index.merge.policy.reclaim_deletes_weight: 3.0
index.merge.policy.max_merge_at_once: 5
index.merge.policy.segments_per_tier: 5

With those in place, the deleted document percent hovers around 15% rather
than shooting up to 45% like it used to.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87cd5fa0-1af3-4bbe-8565-5fbc921d3c2e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Good to know, thanks Dan.
I think you could have also ... and still can increase index.merge.policy.reclaim_deletes_weight
a bit more to push that 15% down a little further.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wednesday, December 11, 2013 6:15:34 PM UTC-5, Dan Everton wrote:

Just to follow this up. We've managed to sort out our settings to cope
with the deletion rate we have. The key seemed to be disabling the store
throttle completely. Elasticsearch is running on physical instances with
fast(ish) disks so the throttle just seemed to be getting in the way. The
final settings were

indices.store.throttle.type: none
index.merge.policy.reclaim_deletes_weight: 3.0
index.merge.policy.max_merge_at_once: 5
index.merge.policy.segments_per_tier: 5

With those in place, the deleted document percent hovers around 15% rather
than shooting up to 45% like it used to.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec9d60ad-5a32-4496-9ec4-d4db20ebaa31%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.