How many old indices I should have for ElasticSearch?

Elastic Stack Elasticsearch
1

I am using ElasticSearch with mainly default configuration and noticed recently that my old indices are eating too much space. I believe they are being created automatically because of default configuration since I have not done any such configuration. Please help me with:

  • how many old indices I should generally keep?
  • does my today's search use index created yesterday?
  • can I live with just one index copy and reuse the same every time?
  • what purpose old indices serve?
  • where is the setting because of which new indices are getting created automatically every day?

Since ES is running on a Production server, I simply cannot delete old indices hence need expert advice. Thanks.

PS: SO link for this question: http://stackoverflow.com/questions/32933087/how-many-old-indices-i-should-have-for-elasticsearch

2

Are you using time-series indexes created by e.g. Logstash? Or what are these "old indexes" you speak of? Please explain what kind of data you're storing and how it's submitted into ES.

3

I am seeing multiple index folders in "/var/lib/elasticsearch/elasticsearch/nodes/0/indices/" folder:

.marvel-2015.09.08/ .marvel-2015.09.16/ .marvel-2015.09.24/ .marvel-2015.10.02/
.marvel-2015.09.01/ .marvel-2015.09.09/ .marvel-2015.09.17/ .marvel-2015.09.25/ .marvel-2015.10.03/
.marvel-2015.09.02/ .marvel-2015.09.10/ .marvel-2015.09.18/ .marvel-2015.09.26/ .marvel-2015.10.04/
.marvel-2015.09.03/ .marvel-2015.09.11/ .marvel-2015.09.19/ .marvel-2015.09.27/ .marvel-kibana/
.marvel-2015.09.04/ .marvel-2015.09.12/ .marvel-2015.09.20/ .marvel-2015.09.28/
.marvel-2015.09.05/ .marvel-2015.09.13/ .marvel-2015.09.21/ .marvel-2015.09.29/
.marvel-2015.09.06/ .marvel-2015.09.14/ .marvel-2015.09.22/ .marvel-2015.09.30/
.marvel-2015.09.07/ .marvel-2015.09.15/ .marvel-2015.09.23/ .marvel-2015.10.01/

They still collectively eat 11GB after I deleted folders specific to the August month.

I am storing textual records (2000 only currently) in the index which are inserted/updated/deleted through CodeIgniter ES library. CodeIgniter config is as follows:

$param = array(
    'connectionClass'       => '\Elasticsearch\Connections\GuzzleConnection',
    'connectionFactoryClass'=> '\Elasticsearch\Connections\ConnectionFactory',
    'connectionPoolClass'   => '\Elasticsearch\ConnectionPool\StaticNoPingConnectionPool',
    'selectorClass'         => '\Elasticsearch\ConnectionPool\Selectors\RoundRobinSelector',
    'serializerClass'       => '\Elasticsearch\Serializers\SmartSerializer',
    'sniffOnStart'          => false,
    'connectionParams'      => array(),
    'logging'               => false,
    'logObject'             => null,
    'logPath'               => 'elasticsearch.log',
    
    'traceObject'           => null,
    'tracePath'             => 'elasticsearch.log',

    'guzzleOptions'         => array(),
    'connectionPoolParams'  => array(
        'randomizeHosts' => true
    ),
    'retries'               => null
);
$params['logging'] = true;
$params['logPath'] = '/var/log/elasticsearch/elasticsearch.log';
$params['logPermission'] = 0664;

$client = new Elasticsearch\Client($params);

I don't use Logstash and don't know why the indices are getting created automatically every day. I could not find any such setting in my config files.

Thanks for your reply.

4

Oh, Marvel indexes. That's Elasticsearch performance metrics stored by the Marvel add-on. You only need to keep those indexes for as long as you need the data for performance analysis or whatever it is that you use them for. Create a cron job that calls Curator to delete indexes after they reach a certain age.

5

Ohh, I got it now. I was always wondering why my tiny dataset is taking space in GBs.

I deleted all marvel indices and turned off logging as well by adding this my .yml file:

marvel.agent.enabled = false

Thanks for saving my day and all your support.

6

Never delete indices from the FS like that, always use a delete call via the API.

7

Yes, I know that. I have deleted old indices using the DELETE call only.

8

Great! :slight_smile: