I have many deleted documents in my index and I'd like to clear them so
that df/ttf statistics aren't skewed by these deleted docs.
I try the following command :
curl -XPOST ' http://localhost:9200/myindex/_optimize?only_expunge_deletes=true'
But the number of deleted documents is still the same, as I can see using
head plugin.
Ideally the enitre set of deleted documents can would be only removed when
you execute a full merge into 1 segment.
But then this eats lots of time if you have a large index.
On using only_expunge_deletes , you are suggesting ES to prioritize those
segments which have higher number of deleted documents while merging. But
then if the merge policy feels it doesnt need to merge any more document ,
it wont perform it.
I have many deleted documents in my index and I'd like to clear them so
that df/ttf statistics aren't skewed by these deleted docs.
I try the following command :
curl -XPOST 'http://localhost:9200/myindex/_optimize?only_
expunge_deletes=true'
But the number of deleted documents is still the same, as I can see using
head plugin.
Thanks for your answer.
So I understand that the only way to surely clear all deleted documents is
to use max_num_documents=1 and indeed in my case, this takes too much time
to be launched often.
In fact my problem is that it seems df statistic is calculated among all
docs, including deleted docs.
Do you know if there is a way to get it calculated only among docs that are
not deleted ?
Thanks
Le vendredi 13 mars 2015 12:58:16 UTC+1, vineeth mohan a écrit :
Hi ,
Ideally the enitre set of deleted documents can would be only removed
when you execute a full merge into 1 segment.
But then this eats lots of time if you have a large index.
On using only_expunge_deletes , you are suggesting ES to prioritize those
segments which have higher number of deleted documents while merging. But
then if the merge policy feels it doesnt need to merge any more document ,
it wont perform it.
On Fri, Mar 13, 2015 at 5:05 PM, Christophe Rosko <chris...@digiteka.com
<javascript:>> wrote:
Hi !
I have many deleted documents in my index and I'd like to clear them so
that df/ttf statistics aren't skewed by these deleted docs.
I try the following command :
curl -XPOST 'http://localhost:9200/myindex/_optimize?only_
expunge_deletes=true'
But the number of deleted documents is still the same, as I can see using
head plugin.
Note that only_expunge_deletes=true will only merge the segment away if it
has > 10% delete docs by default, otherwise it leaves the segment as is.
See
on how to change that 10% default.
But it's almost always unproductive to fret about deleted docs; see this
blog post I wrote recently:
Also the docFreq stat for a given term will always include the deleted docs
but this is usually not a problem unless the docs you've deleted vary
drastically statistically from the remaining docs.
Thanks for your answer.
So I understand that the only way to surely clear all deleted documents is
to use max_num_documents=1 and indeed in my case, this takes too much time
to be launched often.
In fact my problem is that it seems df statistic is calculated among all
docs, including deleted docs.
Do you know if there is a way to get it calculated only among docs that
are not deleted ?
Thanks
Le vendredi 13 mars 2015 12:58:16 UTC+1, vineeth mohan a écrit :
Hi ,
Ideally the enitre set of deleted documents can would be only removed
when you execute a full merge into 1 segment.
But then this eats lots of time if you have a large index.
On using only_expunge_deletes , you are suggesting ES to prioritize those
segments which have higher number of deleted documents while merging. But
then if the merge policy feels it doesnt need to merge any more document ,
it wont perform it.
I have many deleted documents in my index and I'd like to clear them so
that df/ttf statistics aren't skewed by these deleted docs.
I try the following command :
curl -XPOST 'http://localhost:9200/myindex/_optimize?only_expunge_
deletes=true'
But the number of deleted documents is still the same, as I can see
using head plugin.
As I have lots of updates only on recent docs, terms that have been
recently indexed have their docFreq really impacted for a while.
I'll keep thinking about a way to limit the impact on docFreqs and not
worry too much about deleted docs.
Thanks for your answer and your post, it made things much clearer for me.
Christophe
Le vendredi 13 mars 2015 14:03:45 UTC+1, Michael McCandless a écrit :
Note that only_expunge_deletes=true will only merge the segment away if it
has > 10% delete docs by default, otherwise it leaves the segment as is.
Also the docFreq stat for a given term will always include the deleted
docs but this is usually not a problem unless the docs you've deleted vary
drastically statistically from the remaining docs.
On Fri, Mar 13, 2015 at 5:20 AM, Christophe Rosko <chris...@digiteka.com
<javascript:>> wrote:
Hi Vineeth,
Thanks for your answer.
So I understand that the only way to surely clear all deleted documents
is to use max_num_documents=1 and indeed in my case, this takes too much
time to be launched often.
In fact my problem is that it seems df statistic is calculated among all
docs, including deleted docs.
Do you know if there is a way to get it calculated only among docs that
are not deleted ?
Thanks
Le vendredi 13 mars 2015 12:58:16 UTC+1, vineeth mohan a écrit :
Hi ,
Ideally the enitre set of deleted documents can would be only removed
when you execute a full merge into 1 segment.
But then this eats lots of time if you have a large index.
On using only_expunge_deletes , you are suggesting ES to prioritize
those segments which have higher number of deleted documents while merging.
But then if the merge policy feels it doesnt need to merge any more
document , it wont perform it.
I have many deleted documents in my index and I'd like to clear them so
that df/ttf statistics aren't skewed by these deleted docs.
I try the following command :
curl -XPOST 'http://localhost:9200/myindex/_optimize?only_expunge_
deletes=true'
But the number of deleted documents is still the same, as I can see
using head plugin.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.