Elasticsearch housekeeping

dey_subhro · June 12, 2019, 9:44pm

Hi,

I am using Amazon ES, can someone tell me what is the ideal way of automatically deleting the old indexes from the cluster considering Amazon ES does not provide the ILM features ?

I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?

Also will there be any impact on the memory if the no. of shards increases while I am deleting the documents older than 30 days daily?

warkolm · June 12, 2019, 9:50pm

Elasticsearch Curator should work for you.

You could always upgrade to our Elasticsearch Service as well

dey_subhro · June 13, 2019, 9:09am

HI @warkolm

I think curator will not work in Amazon ES.

Also if you can answer this query mentioned above
"I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?"

theuntergeek · June 13, 2019, 2:22pm

Curator works just fine with AWS Elasticsearch, so long as you are using AWS Elasticsearch version 5.4 or higher.

goodmirek5849 · June 19, 2019, 1:50pm

I would not recommend that. I am using Elastic Cloud and compared to ES running directly on EC2 it is far less reliable, far more expensive. I have to use curator in addition to ILM. Also, ILM shown strange behavior, which nobody was able to explain. For production use, Elastic Cloud and ILM is far less reliable choice than self-hosted or AWS managed ES with curator.

warkolm · June 19, 2019, 10:46pm

Welcome to the community

Are there other threads that you've posted under a different account with these issues that we might be able to take a look at?

gabriel_tessier · June 20, 2019, 1:00am

Hi,

Have a look at this:

gist.github.com

https://gist.github.com/jarpy/f65b24209e0a9240207599a73ce99bc9

requirements.txt

certifi==2016.8.8
elasticsearch-curator==4.0.6
PyYAML==3.11

serverless-curator.py

# Run Elasticsearch Curator from AWS Lambda.
#
# Edit serverless-curator.yaml to define which indices should be purged.
from __future__ import print_function

import certifi
import curator
import yaml
from curator.exceptions import NoIndices
from elasticsearch import Elasticsearch

This file has been truncated. show original

serverless-curator.yaml

---
# Define Elasticsearch Clusters and indices here, to have them periodically
# pruned by Curator.

- name: example logging cluster
  endpoint: https://curator:PASSWORD@LOG-CLUSTER.us-west-1.aws.found.io:9243/
  indices:
    - prefix: logstash-
      days: 365

This file has been truncated. show original

dey_subhro · June 26, 2019, 2:29pm

No @warkolm I haven't posted it anywhere

system · July 24, 2019, 2:29pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.