How does ES handle deletes? (keeping a sliding window of documents)


(Theo) #1

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I haven't
been able to figure out completely is how ES handles deletes. My plan is to
continuously throw new documents into ES, but I only want to keep a sliding
window of a couple of days, so every day I want to get rid of all the
documents I added say a week ago. The documents will have a date so it
doesn't look like there's going to be difficult to actually delete them, a
delete-by query seems to do exactly what I want. However, how will ES
handle that in terms of performance? What effects will it have on long time
performance to always be deleting a big chunk of the data? Some databases
would get heavily fragmented from that kind of operation, while others
handle it gracefully by doing compactions from time to time. How does ES do
it?

As a test I added documents to ES until it used around a gigabyte of disk,
then deleted a big chunk of the database and saw that eventually the size
on disk decreased. That looks promising, but I'd like to be sure that what
I saw was not due to something else.

yours
Theo

--


(Paul Smith) #2

why not just have an index per day, and then create an alias across the
per-day indices. Then you just delete the indices you don't need anymore.
Basically like a 'truncate table' in SQL. you can easily query the REST
api for all the indice names, and if they're named in a good format
"myindex.2012-10-03' you can easily parse that and find a cull list to send
Delete requests.

On 3 October 2012 16:27, Theo iconara@gmail.com wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I
haven't been able to figure out completely is how ES handles deletes. My
plan is to continuously throw new documents into ES, but I only want to
keep a sliding window of a couple of days, so every day I want to get rid
of all the documents I added say a week ago. The documents will have a date
so it doesn't look like there's going to be difficult to actually delete
them, a delete-by query seems to do exactly what I want. However, how will
ES handle that in terms of performance? What effects will it have on long
time performance to always be deleting a big chunk of the data? Some
databases would get heavily fragmented from that kind of operation, while
others handle it gracefully by doing compactions from time to time. How
does ES do it?

As a test I added documents to ES until it used around a gigabyte of disk,
then deleted a big chunk of the database and saw that eventually the size
on disk decreased. That looks promising, but I'd like to be sure that what
I saw was not due to something else.

yours
Theo

--

--


(David Pilato) #3

You can use TTL field feature http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html
Or you can create an index per week and add alias on this index.

Let's say you create each monday an index named myindex201234 (2012 is the year and 34 the week number) and then you define an alias named myindex on myindex201234.
The following week, you will add a new index myindex201235 and define myindex as alias.

So, when you query for myindex you get all your docs.

Deleting a week is only deleting an index which is really fast when comparing to deletebyquery.
http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:27, Theo iconara@gmail.com a écrit :

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I haven't been able to figure out completely is how ES handles deletes. My plan is to continuously throw new documents into ES, but I only want to keep a sliding window of a couple of days, so every day I want to get rid of all the documents I added say a week ago. The documents will have a date so it doesn't look like there's going to be difficult to actually delete them, a delete-by query seems to do exactly what I want. However, how will ES handle that in terms of performance? What effects will it have on long time performance to always be deleting a big chunk of the data? Some databases would get heavily fragmented from that kind of operation, while others handle it gracefully by doing compactions from time to time. How does ES do it?

As a test I added documents to ES until it used around a gigabyte of disk, then deleted a big chunk of the database and saw that eventually the size on disk decreased. That looks promising, but I'd like to be sure that what I saw was not due to something else.

yours
Theo

--


(David Pilato) #4

Paul was faster than me :wink:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:41, Paul Smith tallpsmith@gmail.com a écrit :

why not just have an index per day, and then create an alias across the per-day indices. Then you just delete the indices you don't need anymore. Basically like a 'truncate table' in SQL. you can easily query the REST api for all the indice names, and if they're named in a good format "myindex.2012-10-03' you can easily parse that and find a cull list to send Delete requests.

On 3 October 2012 16:27, Theo iconara@gmail.com wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I haven't been able to figure out completely is how ES handles deletes. My plan is to continuously throw new documents into ES, but I only want to keep a sliding window of a couple of days, so every day I want to get rid of all the documents I added say a week ago. The documents will have a date so it doesn't look like there's going to be difficult to actually delete them, a delete-by query seems to do exactly what I want. However, how will ES handle that in terms of performance? What effects will it have on long time performance to always be deleting a big chunk of the data? Some databases would get heavily fragmented from that kind of operation, while others handle it gracefully by doing compactions from time to time. How does ES do it?

As a test I added documents to ES until it used around a gigabyte of disk, then deleted a big chunk of the database and saw that eventually the size on disk decreased. That looks promising, but I'd like to be sure that what I saw was not due to something else.

yours
Theo

--

--


(Theo) #5

Thanks! The idea of keeping an index per day or week makes sense. We've
done something like that in other databases when we've had to keep sliding
windows, but it's always a hassle because you have to add a layer that maps
your queries against the right database depending on the date in the query
and so on. Index aliases seem to solve that very nicely -- I wasn't aware
of them but now I'll go read all about it.

T#

On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote:

Paul was faster than me :wink:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:41, Paul Smith <tallp...@gmail.com <javascript:>> a
écrit :

why not just have an index per day, and then create an alias across the
per-day indices. Then you just delete the indices you don't need anymore.
Basically like a 'truncate table' in SQL. you can easily query the REST
api for all the indice names, and if they're named in a good format
"myindex.2012-10-03' you can easily parse that and find a cull list to send
Delete requests.

On 3 October 2012 16:27, Theo <ico...@gmail.com <javascript:>> wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I
haven't been able to figure out completely is how ES handles deletes. My
plan is to continuously throw new documents into ES, but I only want to
keep a sliding window of a couple of days, so every day I want to get rid
of all the documents I added say a week ago. The documents will have a date
so it doesn't look like there's going to be difficult to actually delete
them, a delete-by query seems to do exactly what I want. However, how will
ES handle that in terms of performance? What effects will it have on long
time performance to always be deleting a big chunk of the data? Some
databases would get heavily fragmented from that kind of operation, while
others handle it gracefully by doing compactions from time to time. How
does ES do it?

As a test I added documents to ES until it used around a gigabyte of
disk, then deleted a big chunk of the database and saw that eventually the
size on disk decreased. That looks promising, but I'd like to be sure that
what I saw was not due to something else.

yours
Theo

--

--

--


(Lukáš Vlček) #6

Check also this presentation
http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

Shay mentioned this sliding window ("time" data flow) strategy in his
presentation.

Regards,
Lukas

On Wed, Oct 3, 2012 at 9:06 AM, Theo iconara@gmail.com wrote:

Thanks! The idea of keeping an index per day or week makes sense. We've
done something like that in other databases when we've had to keep sliding
windows, but it's always a hassle because you have to add a layer that maps
your queries against the right database depending on the date in the query
and so on. Index aliases seem to solve that very nicely -- I wasn't aware
of them but now I'll go read all about it.

T#

On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote:

Paul was faster than me :wink:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:41, Paul Smith tallp...@gmail.com a écrit :

why not just have an index per day, and then create an alias across the
per-day indices. Then you just delete the indices you don't need anymore.
Basically like a 'truncate table' in SQL. you can easily query the REST
api for all the indice names, and if they're named in a good format
"myindex.2012-10-03' you can easily parse that and find a cull list to send
Delete requests.

On 3 October 2012 16:27, Theo ico...@gmail.com wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I
haven't been able to figure out completely is how ES handles deletes. My
plan is to continuously throw new documents into ES, but I only want to
keep a sliding window of a couple of days, so every day I want to get rid
of all the documents I added say a week ago. The documents will have a date
so it doesn't look like there's going to be difficult to actually delete
them, a delete-by query seems to do exactly what I want. However, how will
ES handle that in terms of performance? What effects will it have on long
time performance to always be deleting a big chunk of the data? Some
databases would get heavily fragmented from that kind of operation, while
others handle it gracefully by doing compactions from time to time. How
does ES do it?

As a test I added documents to ES until it used around a gigabyte of
disk, then deleted a big chunk of the database and saw that eventually the
size on disk decreased. That looks promising, but I'd like to be sure that
what I saw was not due to something else.

yours
Theo

--

--

--

--


(Raffaele Sena) #7

I just realized ES also support a TTL on documents
(http://www.elasticsearch.org/guide/reference/api/index_.html look for
TTL).

Unless you really want to delete old documents only when you add new
ones, this may be what you are looking for.

-- Raffaele

On Wed, Oct 3, 2012 at 12:20 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Check also this presentation
http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

Shay mentioned this sliding window ("time" data flow) strategy in his
presentation.

Regards,
Lukas

On Wed, Oct 3, 2012 at 9:06 AM, Theo iconara@gmail.com wrote:

Thanks! The idea of keeping an index per day or week makes sense. We've
done something like that in other databases when we've had to keep sliding
windows, but it's always a hassle because you have to add a layer that maps
your queries against the right database depending on the date in the query
and so on. Index aliases seem to solve that very nicely -- I wasn't aware of
them but now I'll go read all about it.

T#

On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote:

Paul was faster than me :wink:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:41, Paul Smith tallp...@gmail.com a écrit :

why not just have an index per day, and then create an alias across the
per-day indices. Then you just delete the indices you don't need anymore.
Basically like a 'truncate table' in SQL. you can easily query the REST api
for all the indice names, and if they're named in a good format
"myindex.2012-10-03' you can easily parse that and find a cull list to send
Delete requests.

On 3 October 2012 16:27, Theo ico...@gmail.com wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I
haven't been able to figure out completely is how ES handles deletes. My
plan is to continuously throw new documents into ES, but I only want to keep
a sliding window of a couple of days, so every day I want to get rid of all
the documents I added say a week ago. The documents will have a date so it
doesn't look like there's going to be difficult to actually delete them, a
delete-by query seems to do exactly what I want. However, how will ES handle
that in terms of performance? What effects will it have on long time
performance to always be deleting a big chunk of the data? Some databases
would get heavily fragmented from that kind of operation, while others
handle it gracefully by doing compactions from time to time. How does ES do
it?

As a test I added documents to ES until it used around a gigabyte of
disk, then deleted a big chunk of the database and saw that eventually the
size on disk decreased. That looks promising, but I'd like to be sure that
what I saw was not due to something else.

yours
Theo

--

--

--

--

--


(Theo) #8

TTL is very convenient, but it comes back to my original question: how does
the underlying database handle deletions of lots of documents? will it
become fragmented over time and get slower? this happens for example in
MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate.
Cassandra does not suffer from this effect because it continuously compacts
its on-disk data. how does ES perform a delete? does it create a hole in
the on-disk datastructures that will later be filled by new data, or does
it add a tombstone that will get compacted away at a later time, or does it
do something else? (I guess the question might actually be how Lucene does
this).

I think that the one-index-per-week solution will work great, but if ES
handles deletions gracefully without getting fragmented that would be very
good too.

T#

On Thursday, October 4, 2012 12:42:26 AM UTC+2, Raffaele Sena wrote:

I just realized ES also support a TTL on documents
(http://www.elasticsearch.org/guide/reference/api/index_.html look for
TTL).

Unless you really want to delete old documents only when you add new
ones, this may be what you are looking for.

-- Raffaele

On Wed, Oct 3, 2012 at 12:20 AM, Lukáš Vlček <lukas...@gmail.com<javascript:>>
wrote:

Check also this presentation

http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html

Shay mentioned this sliding window ("time" data flow) strategy in his
presentation.

Regards,
Lukas

On Wed, Oct 3, 2012 at 9:06 AM, Theo <ico...@gmail.com <javascript:>>
wrote:

Thanks! The idea of keeping an index per day or week makes sense. We've
done something like that in other databases when we've had to keep
sliding

windows, but it's always a hassle because you have to add a layer that
maps

your queries against the right database depending on the date in the
query

and so on. Index aliases seem to solve that very nicely -- I wasn't
aware of

them but now I'll go read all about it.

T#

On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote:

Paul was faster than me :wink:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2012 à 08:41, Paul Smith tallp...@gmail.com a écrit :

why not just have an index per day, and then create an alias across
the

per-day indices. Then you just delete the indices you don't need
anymore.

Basically like a 'truncate table' in SQL. you can easily query the
REST api

for all the indice names, and if they're named in a good format
"myindex.2012-10-03' you can easily parse that and find a cull list to
send

Delete requests.

On 3 October 2012 16:27, Theo ico...@gmail.com wrote:

Hi,

I'm evaluating ElasticSearch for a new thing I'm working on. What I
haven't been able to figure out completely is how ES handles deletes.
My

plan is to continuously throw new documents into ES, but I only want
to keep

a sliding window of a couple of days, so every day I want to get rid
of all

the documents I added say a week ago. The documents will have a date
so it

doesn't look like there's going to be difficult to actually delete
them, a

delete-by query seems to do exactly what I want. However, how will ES
handle

that in terms of performance? What effects will it have on long time
performance to always be deleting a big chunk of the data? Some
databases

would get heavily fragmented from that kind of operation, while
others

handle it gracefully by doing compactions from time to time. How does
ES do

it?

As a test I added documents to ES until it used around a gigabyte of
disk, then deleted a big chunk of the database and saw that
eventually the

size on disk decreased. That looks promising, but I'd like to be sure
that

what I saw was not due to something else.

yours
Theo

--

--

--

--

--


(Paul Smith) #9

On 4 October 2012 16:17, Theo iconara@gmail.com wrote:

TTL is very convenient, but it comes back to my original question: how
does the underlying database handle deletions of lots of documents? will it
become fragmented over time and get slower? this happens for example in
MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate.
Cassandra does not suffer from this effect because it continuously compacts
its on-disk data. how does ES perform a delete? does it create a hole in
the on-disk datastructures that will later be filled by new data, or does
it add a tombstone that will get compacted away at a later time, or does it
do something else? (I guess the question might actually be how Lucene does
this).

I think that the one-index-per-week solution will work great, but if ES
handles deletions gracefully without getting fragmented that would be very
good too.

As I understand how Lucene works, a delete marks a specific Document ID (a
Lucene internal term) as deleted. Each segment maintains a delete marker
(*.del files for each segment in the indices directory of a shard). This
makes it a fast delete, because no real index modification needs to be made
other than writing the ID to be deleted (that file is mapped into memory
too). When a search happens, all matching Doc ID's form part of a matching
bitmask for the matches, which then get the deletes applied to them by
masking out the document Ids of 'matches' that are no longer there.
Obviously in a newly created index with no mutations, the delete file is
empty, so this is a no-op, so there's a slight overhead when theres deletes
to be applied.

Since an 'update' in Lucene is really a delete and and add operation, these
deletes accumulate over time, but as new items are added, merges are
triggered so that the # and size of segment files doesn't get too crazy.
During the merge process the deleted records are wiped out during the
merge compaction.

So wholesale deletes by wiping out large segments of your document space
will end up with lots of items in the individual segment delete files,
which isn't terrible, just not optimal. There is potential depending on
the distribution of your documents and their contents on how the term
frequencies etc are, that the actual searches may be slower because there's
all these redundant document Ids in each terms vector (basically wasted
space) that end up getting nulled out, so you do pay a penalty for them,
but it's not a massive overhead.

One can optimize after you bulk delete of course by using the _optimize API
call to compact/merge segments together to remove all these, but it's a lot
of IO for large indices if you've blown a way a huge chunk, and way slower
than a file delete that ends up happening when you _delete the specific
indice. I like to think of it as the difference between an SQL "DELETE"
and "TRUNCATE", the former being 'logged for recovery' (ala in the .del
file, not realyl recoverable, I'm not sure what happens if you manually
remove that .del file) while the latter is much faster.

So, deleting an index wholesale is way more efficient in Lucene world if
you can design your indices & aliases that way, but searches are still
pretty good if you do block deletes within an index, just not as good as
if the index is in a clean state, you'll waste memory and disk head seeks
over time, and you want searching to be fast right? That's why you picked
or are looking at ElasticSearch! :slight_smile:

hopefully that helps

Paul

--


(Theo) #10

Thanks Paul, that was exactly the explanation I was looking for. Now I feel
I have a much better understanding of the situation. For my particular use
case I think partitioning is the way to go (i.e. a new index per
week/day/month/whatever), but it's great to know how ES would handle TTL
and individual deletes, and that it would probably work that way too, just
not as efficiently.

T#

On Thursday, October 4, 2012 9:01:03 AM UTC+2, tallpsmith wrote:

On 4 October 2012 16:17, Theo <ico...@gmail.com <javascript:>> wrote:

TTL is very convenient, but it comes back to my original question: how
does the underlying database handle deletions of lots of documents? will it
become fragmented over time and get slower? this happens for example in
MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate.
Cassandra does not suffer from this effect because it continuously compacts
its on-disk data. how does ES perform a delete? does it create a hole in
the on-disk datastructures that will later be filled by new data, or does
it add a tombstone that will get compacted away at a later time, or does it
do something else? (I guess the question might actually be how Lucene does
this).

I think that the one-index-per-week solution will work great, but if ES
handles deletions gracefully without getting fragmented that would be very
good too.

As I understand how Lucene works, a delete marks a specific Document ID (a
Lucene internal term) as deleted. Each segment maintains a delete marker
(*.del files for each segment in the indices directory of a shard). This
makes it a fast delete, because no real index modification needs to be made
other than writing the ID to be deleted (that file is mapped into memory
too). When a search happens, all matching Doc ID's form part of a matching
bitmask for the matches, which then get the deletes applied to them by
masking out the document Ids of 'matches' that are no longer there.
Obviously in a newly created index with no mutations, the delete file is
empty, so this is a no-op, so there's a slight overhead when theres deletes
to be applied.

Since an 'update' in Lucene is really a delete and and add operation,
these deletes accumulate over time, but as new items are added, merges are
triggered so that the # and size of segment files doesn't get too crazy.
During the merge process the deleted records are wiped out during the
merge compaction.

So wholesale deletes by wiping out large segments of your document space
will end up with lots of items in the individual segment delete files,
which isn't terrible, just not optimal. There is potential depending on
the distribution of your documents and their contents on how the term
frequencies etc are, that the actual searches may be slower because there's
all these redundant document Ids in each terms vector (basically wasted
space) that end up getting nulled out, so you do pay a penalty for them,
but it's not a massive overhead.

One can optimize after you bulk delete of course by using the _optimize
API call to compact/merge segments together to remove all these, but it's a
lot of IO for large indices if you've blown a way a huge chunk, and way
slower than a file delete that ends up happening when you _delete the
specific indice. I like to think of it as the difference between an SQL
"DELETE" and "TRUNCATE", the former being 'logged for recovery' (ala in the
.del file, not realyl recoverable, I'm not sure what happens if you
manually remove that .del file) while the latter is much faster.

So, deleting an index wholesale is way more efficient in Lucene world if
you can design your indices & aliases that way, but searches are still
pretty good if you do block deletes within an index, just not as good as
if the index is in a clean state, you'll waste memory and disk head seeks
over time, and you want searching to be fast right? That's why you picked
or are looking at ElasticSearch! :slight_smile:

hopefully that helps

Paul

--


(system) #11