Elasticsearch is using more and more space on my disc

Elastic Stack Elasticsearch
1

Hello,

I'm using elasticsearch 0.16.2 on a single node. I'm using it to store
almost the same number of document at all time, which are actually a
big event list of the last 7 days.
Every night I purge the database using something like:

curl -XDELETE 'myserver/_query' -d '{
"range" : { "date" : { "lte": "'2011-08-05T00:00:00" } }
}'

Then, I run an optimize query: something like :

curl -XPOST 'myserver/_optimize' -d '{
"max_num_segments" : 1
}

Every seems to work, BUT the data directory of elasticsearch is still
increasing. Is anyone can tell how can I do my document rotation
without always increasing the used space ?

Thanks !

2

Hi Erwan,

You should consider using index aliases for that need.

Create an index each day with the name myserver_yyyymmdd
Create an alias on each daily index called myserver

Then you can make your searches on /myserver

When you have to purge one entire day, just drop the indexmyserver_yyyymmdd
It will take some milliseconds and will really clean your filesystem

Hope this helps
David.

Le 12 août 2011 à 10:39, erlo erwan.loaec@cgin.fr a écrit :

Hello,

I'm using elasticsearch 0.16.2 on a single node. I'm using it to store
almost the same number of document at all time, which are actually a
big event list of the last 7 days.
Every night I purge the database using something like:

curl -XDELETE 'myserver/_query' -d '{
"range" : { "date" : { "lte": "'2011-08-05T00:00:00" } }
}'

Then, I run an optimize query: something like :

curl -XPOST 'myserver/_optimize' -d '{
"max_num_segments" : 1
}

Every seems to work, BUT the data directory of elasticsearch is still
increasing. Is anyone can tell how can I do my document rotation
without always increasing the used space ?

Thanks !

3

Thank you for answer. I will seriously take a look about this in the
future improvement of my application. But, for the moment I have to
keep my current design...
But I think there should have a solution to "clean" the data
filesystem, to reduce size on my disc ?

On 12 août, 10:55, "da...@pilato.fr" da...@pilato.fr wrote:

Hi Erwan,

You should consider using index aliases for that need.

Create an index each day with the name myserver_yyyymmdd
Create an alias on each daily index called myserver

Then you can make your searches on /myserver

When you have to purge one entire day, just drop the indexmyserver_yyyymmdd
It will take some milliseconds and will really clean your filesystem

Hope this helps
David.

Le 12 août 2011 à 10:39, erlo erwan.lo...@cgin.fr a écrit :

Hello,

I'm using elasticsearch 0.16.2 on a single node. I'm using it to store
almost the same number of document at all time, which are actually a
big event list of the last 7 days.
Every night I purge the database using something like:

curl -XDELETE 'myserver/_query' -d '{
"range" : { "date" : { "lte": "'2011-08-05T00:00:00" } }
}'

Then, I run an optimize query: something like :

curl -XPOST 'myserver/_optimize' -d '{
"max_num_segments" : 1
}

Every seems to work, BUT the data directory of elasticsearch is still
increasing. Is anyone can tell how can I do my document rotation
without always increasing the used space ?

Thanks !

4

The max_num_segments is a request parameter:
Elasticsearch Platform — Find real-time answers at scale | Elastic,
Meaning, you ened to run: /_optimize?max_num_segments=1. In 0.16.2 make sure
to flush before as well.

On Fri, Aug 12, 2011 at 12:17 PM, erlo erwan.loaec@cgin.fr wrote:

Thank you for answer. I will seriously take a look about this in the
future improvement of my application. But, for the moment I have to
keep my current design...
But I think there should have a solution to "clean" the data
filesystem, to reduce size on my disc ?

On 12 août, 10:55, "da...@pilato.fr" da...@pilato.fr wrote:

Hi Erwan,

You should consider using index aliases for that need.

Create an index each day with the name myserver_yyyymmdd
Create an alias on each daily index called myserver

Then you can make your searches on /myserver

When you have to purge one entire day, just drop the
indexmyserver_yyyymmdd
It will take some milliseconds and will really clean your filesystem

Hope this helps
David.

Le 12 août 2011 à 10:39, erlo erwan.lo...@cgin.fr a écrit :

Hello,

I'm using elasticsearch 0.16.2 on a single node. I'm using it to store
almost the same number of document at all time, which are actually a
big event list of the last 7 days.
Every night I purge the database using something like:

curl -XDELETE 'myserver/_query' -d '{
"range" : { "date" : { "lte": "'2011-08-05T00:00:00" } }
}'

Then, I run an optimize query: something like :

curl -XPOST 'myserver/_optimize' -d '{
"max_num_segments" : 1
}

Every seems to work, BUT the data directory of elasticsearch is still
increasing. Is anyone can tell how can I do my document rotation
without always increasing the used space ?

Thanks !

5

Thanks a lot !!
My problem was here.

Shay Banon wrote:

The max_num_segments is a request
parameter: Elasticsearch Platform — Find real-time answers at scale | Elastic,
Meaning, you ened to run: /_optimize?max_num_segments=1. In 0.16.2 make
sure to flush before as well.

On Fri, Aug 12, 2011 at 12:17 PM, erlo <erwan.loaec@cgin.fr
mailto:erwan.loaec@cgin.fr> wrote:

Thank you for answer. I will seriously take a look about this in the
future improvement of my application. But, for the moment I have to
keep my current design...
But I think there should have a solution to "clean" the data
filesystem, to reduce size on my disc ?

On 12 août, 10:55, "da...@pilato.fr <mailto:da...@pilato.fr>"
<da...@pilato.fr <mailto:da...@pilato.fr>> wrote:
 > Hi Erwan,
 >  
 >  
 > You should consider using index aliases for that need.
 >  
 > Create an index each day with the name myserver_yyyymmdd
 > Create an alias on each daily index called myserver
 >  
 > Then you can make your searches on /myserver
 >  
 > When you have to purge one entire day, just drop the
indexmyserver_yyyymmdd
 > It will take some milliseconds and will really clean your filesystem
 >  
 >  
 > Hope this helps
 > David. 
 >  
 >  
 >  
 >
 > Le 12 août 2011 à 10:39, erlo <erwan.lo...@cgin.fr
<mailto:erwan.lo...@cgin.fr>> a écrit :
 >
 > > Hello,
 >
 > > I'm using elasticsearch 0.16.2 on a single node. I'm using it
to store
 > > almost the same number of document at all time, which are
actually a
 > > big event list of the last 7 days.
 > > Every night I purge the database using something like:
 >
 > > curl -XDELETE 'myserver/_query' -d '{
 > >     "range" : { "date" : { "lte": "'2011-08-05T00:00:00" } }
 > > }'
 >
 > > Then, I run an optimize query: something like :
 >
 > > curl -XPOST 'myserver/_optimize' -d '{
 > >     "max_num_segments" : 1
 > > }
 >
 > > Every seems to work, BUT the data directory of elasticsearch is
still
 > > increasing. Is anyone can tell how can I do my document rotation
 > > without always increasing the used space ?
 >
 > > Thanks !