On 4 October 2012 16:17, Theo iconara@gmail.com wrote:
TTL is very convenient, but it comes back to my original question: how
does the underlying database handle deletions of lots of documents? will it
become fragmented over time and get slower? this happens for example in
MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate.
Cassandra does not suffer from this effect because it continuously compacts
its on-disk data. how does ES perform a delete? does it create a hole in
the on-disk datastructures that will later be filled by new data, or does
it add a tombstone that will get compacted away at a later time, or does it
do something else? (I guess the question might actually be how Lucene does
this).
I think that the one-index-per-week solution will work great, but if ES
handles deletions gracefully without getting fragmented that would be very
good too.
As I understand how Lucene works, a delete marks a specific Document ID (a
Lucene internal term) as deleted. Each segment maintains a delete marker
(*.del files for each segment in the indices directory of a shard). This
makes it a fast delete, because no real index modification needs to be made
other than writing the ID to be deleted (that file is mapped into memory
too). When a search happens, all matching Doc ID's form part of a matching
bitmask for the matches, which then get the deletes applied to them by
masking out the document Ids of 'matches' that are no longer there.
Obviously in a newly created index with no mutations, the delete file is
empty, so this is a no-op, so there's a slight overhead when theres deletes
to be applied.
Since an 'update' in Lucene is really a delete and and add operation, these
deletes accumulate over time, but as new items are added, merges are
triggered so that the # and size of segment files doesn't get too crazy.
During the merge process the deleted records are wiped out during the
merge compaction.
So wholesale deletes by wiping out large segments of your document space
will end up with lots of items in the individual segment delete files,
which isn't terrible, just not optimal. There is potential depending on
the distribution of your documents and their contents on how the term
frequencies etc are, that the actual searches may be slower because there's
all these redundant document Ids in each terms vector (basically wasted
space) that end up getting nulled out, so you do pay a penalty for them,
but it's not a massive overhead.
One can optimize after you bulk delete of course by using the _optimize API
call to compact/merge segments together to remove all these, but it's a lot
of IO for large indices if you've blown a way a huge chunk, and way slower
than a file delete that ends up happening when you _delete the specific
indice. I like to think of it as the difference between an SQL "DELETE"
and "TRUNCATE", the former being 'logged for recovery' (ala in the .del
file, not realyl recoverable, I'm not sure what happens if you manually
remove that .del file) while the latter is much faster.
So, deleting an index wholesale is way more efficient in Lucene world if
you can design your indices & aliases that way, but searches are still
pretty good if you do block deletes within an index, just not as good as
if the index is in a clean state, you'll waste memory and disk head seeks
over time, and you want searching to be fast right? That's why you picked
or are looking at Elasticsearch! 
hopefully that helps
Paul
--
