How many old indices I should have for ElasticSearch?

(Anuj Aggarwal) #1

I am using ElasticSearch with mainly default configuration and noticed recently that my old indices are eating too much space. I believe they are being created automatically because of default configuration since I have not done any such configuration. Please help me with:

  • how many old indices I should generally keep?
  • does my today's search use index created yesterday?
  • can I live with just one index copy and reuse the same every time?
  • what purpose old indices serve?
  • where is the setting because of which new indices are getting created automatically every day?

Since ES is running on a Production server, I simply cannot delete old indices hence need expert advice. Thanks.

PS: SO link for this question:

(Magnus Bäck) #2

Are you using time-series indexes created by e.g. Logstash? Or what are these "old indexes" you speak of? Please explain what kind of data you're storing and how it's submitted into ES.

(Anuj Aggarwal) #3

I am seeing multiple index folders in "/var/lib/elasticsearch/elasticsearch/nodes/0/indices/" folder:

.marvel-2015.09.08/ .marvel-2015.09.16/ .marvel-2015.09.24/ .marvel-2015.10.02/
.marvel-2015.09.01/ .marvel-2015.09.09/ .marvel-2015.09.17/ .marvel-2015.09.25/ .marvel-2015.10.03/
.marvel-2015.09.02/ .marvel-2015.09.10/ .marvel-2015.09.18/ .marvel-2015.09.26/ .marvel-2015.10.04/
.marvel-2015.09.03/ .marvel-2015.09.11/ .marvel-2015.09.19/ .marvel-2015.09.27/ .marvel-kibana/
.marvel-2015.09.04/ .marvel-2015.09.12/ .marvel-2015.09.20/ .marvel-2015.09.28/
.marvel-2015.09.05/ .marvel-2015.09.13/ .marvel-2015.09.21/ .marvel-2015.09.29/
.marvel-2015.09.06/ .marvel-2015.09.14/ .marvel-2015.09.22/ .marvel-2015.09.30/
.marvel-2015.09.07/ .marvel-2015.09.15/ .marvel-2015.09.23/ .marvel-2015.10.01/

They still collectively eat 11GB after I deleted folders specific to the August month.

I am storing textual records (2000 only currently) in the index which are inserted/updated/deleted through CodeIgniter ES library. CodeIgniter config is as follows:

$param = array(
    'connectionClass'       => '\Elasticsearch\Connections\GuzzleConnection',
    'connectionFactoryClass'=> '\Elasticsearch\Connections\ConnectionFactory',
    'connectionPoolClass'   => '\Elasticsearch\ConnectionPool\StaticNoPingConnectionPool',
    'selectorClass'         => '\Elasticsearch\ConnectionPool\Selectors\RoundRobinSelector',
    'serializerClass'       => '\Elasticsearch\Serializers\SmartSerializer',
    'sniffOnStart'          => false,
    'connectionParams'      => array(),
    'logging'               => false,
    'logObject'             => null,
    'logPath'               => 'elasticsearch.log',
    'traceObject'           => null,
    'tracePath'             => 'elasticsearch.log',

    'guzzleOptions'         => array(),
    'connectionPoolParams'  => array(
        'randomizeHosts' => true
    'retries'               => null
$params['logging'] = true;
$params['logPath'] = '/var/log/elasticsearch/elasticsearch.log';
$params['logPermission'] = 0664;

$client = new Elasticsearch\Client($params);

I don't use Logstash and don't know why the indices are getting created automatically every day. I could not find any such setting in my config files.

Thanks for your reply.

(Magnus Bäck) #4

Oh, Marvel indexes. That's Elasticsearch performance metrics stored by the Marvel add-on. You only need to keep those indexes for as long as you need the data for performance analysis or whatever it is that you use them for. Create a cron job that calls Curator to delete indexes after they reach a certain age.

(Anuj Aggarwal) #5

Ohh, I got it now. I was always wondering why my tiny dataset is taking space in GBs.

I deleted all marvel indices and turned off logging as well by adding this my .yml file:

marvel.agent.enabled = false

Thanks for saving my day and all your support.

(Mark Walkom) #6

Never delete indices from the FS like that, always use a delete call via the API.

(Anuj Aggarwal) #7

Yes, I know that. I have deleted old indices using the DELETE call only.

(Mark Walkom) #8

Great! :slight_smile:

(system) #9