The documents deleted while the size of the index kremains the same after using delete by query?

Dear all,

I found that after I conducted a delete by query operation, the matched
documents in the index have been deleted, while the size of the index
remains the same.

Could anybody tell me why? Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

physical deletions only occur after a lucene merge, which is not executed
immediately. Until then a document is only marked as deleted, but not wiped
off the disk. If you put the same document twice, your disk space will
increase as well until the next merge happens. This lies in the nature of
the append-only mechanism of lucene.

You can use the segment spy plugin to visualize merges. Another good read
is this blog post

Hope this helps...

--Alex

On Tue, Jul 23, 2013 at 11:29 AM, Jingang Wang bitwjg@gmail.com wrote:

Dear all,

I found that after I conducted a delete by query operation, the matched
documents in the index have been deleted, while the size of the index
remains the same.

Could anybody tell me why? Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Alex,

Thanks for your informative reply.

From the post, I have learned that the size of the index would remain the
same until the new merge operation.
If the index has been constructed over and then some documents in it were
deleted, the size would not change because of no further merge execution.
Is there some api could make the merge happens and decrease the size of the
index?
Thanks!

On Tue, Jul 23, 2013 at 5:59 PM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

physical deletions only occur after a lucene merge, which is not executed
immediately. Until then a document is only marked as deleted, but not wiped
off the disk. If you put the same document twice, your disk space will
increase as well until the next merge happens. This lies in the nature of
the append-only mechanism of lucene.

You can use the segment spy plugin to visualize merges. Another good read
is this blog post

http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Hope this helps...

--Alex

On Tue, Jul 23, 2013 at 11:29 AM, Jingang Wang bitwjg@gmail.com wrote:

Dear all,

I found that after I conducted a delete by query operation, the matched
documents in the index have been deleted, while the size of the index
remains the same.

Could anybody tell me why? Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kg4EwfzZZzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Wang Jingang(王金刚)
Ph.D. Candidate at
Lab of High Volume Language Information Processing & Cloud Computing
School of Computer Science
Beijing Institute of Technology
Beijing 100081
P.R China

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

See Optimize API: http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize/

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 24 juil. 2013 à 04:34, Jingang Wang bitwjg@gmail.com a écrit :

Hi Alex,

Thanks for your informative reply.

From the post, I have learned that the size of the index would remain the same until the new merge operation.
If the index has been constructed over and then some documents in it were deleted, the size would not change because of no further merge execution.
Is there some api could make the merge happens and decrease the size of the index?
Thanks!

On Tue, Jul 23, 2013 at 5:59 PM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

physical deletions only occur after a lucene merge, which is not executed immediately. Until then a document is only marked as deleted, but not wiped off the disk. If you put the same document twice, your disk space will increase as well until the next merge happens. This lies in the nature of the append-only mechanism of lucene.

You can use the segment spy plugin to visualize merges. Another good read is this blog post
http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Hope this helps...

--Alex

On Tue, Jul 23, 2013 at 11:29 AM, Jingang Wang bitwjg@gmail.com wrote:

Dear all,

I found that after I conducted a delete by query operation, the matched documents in the index have been deleted, while the size of the index remains the same.

Could anybody tell me why? Thanks in advance!

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/kg4EwfzZZzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Wang Jingang(王金刚)
Ph.D. Candidate at
Lab of High Volume Language Information Processing & Cloud Computing
School of Computer Science
Beijing Institute of Technology
Beijing 100081
P.R China

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David,

That's what I need, thank you!!!

On Wed, Jul 24, 2013 at 12:53 PM, David Pilato david@pilato.fr wrote:

See Optimize API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize/

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 24 juil. 2013 à 04:34, Jingang Wang bitwjg@gmail.com a écrit :

Hi Alex,

Thanks for your informative reply.

From the post, I have learned that the size of the index would remain the
same until the new merge operation.
If the index has been constructed over and then some documents in it were
deleted, the size would not change because of no further merge execution.
Is there some api could make the merge happens and decrease the size of
the index?
Thanks!

On Tue, Jul 23, 2013 at 5:59 PM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

physical deletions only occur after a lucene merge, which is not executed
immediately. Until then a document is only marked as deleted, but not wiped
off the disk. If you put the same document twice, your disk space will
increase as well until the next merge happens. This lies in the nature of
the append-only mechanism of lucene.

You can use the segment spy plugin to visualize merges. Another good read
is this blog post

http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Hope this helps...

--Alex

On Tue, Jul 23, 2013 at 11:29 AM, Jingang Wang bitwjg@gmail.com wrote:

Dear all,

I found that after I conducted a delete by query operation, the matched
documents in the index have been deleted, while the size of the index
remains the same.

Could anybody tell me why? Thanks in advance!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kg4EwfzZZzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Wang Jingang(王金刚)
Ph.D. Candidate at
Lab of High Volume Language Information Processing & Cloud Computing
School of Computer Science
Beijing Institute of Technology
Beijing 100081
P.R China

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kg4EwfzZZzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Wang Jingang(王金刚)
Ph.D. Candidate at
Lab of High Volume Language Information Processing & Cloud Computing
School of Computer Science
Beijing Institute of Technology
Beijing 100081
P.R China

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.