I am new to Elasticsearch. I have a 3 data node and 3 master node elastic cluster deployed in Kubernetes. It was working well until recently there is a large intake of data. Now, I am in a stage where I need to apply index refresh interval configuration to 120s to allow optimal usage of the cluster. I am able to do it on individual index level using this method. but I am not able to do at the cluster level. I have a process that creates a new index everyday and do not have more than 15 indexes totally at any point of time in the cluster. So, currently, I am doing it manually by using the method afore mentioned/using Kibana UI. I tried to do this couple of ways both failed.
- Used the settings PUT method to force the index settings at the global level and it gives an error of no requests in the range
PUT /_cluster/settings -d { "index" : { "refresh_interval" : "120s" } }
- I used the elasticsearch yaml to set this value for the data node and the data node fails to come up.
elasticsearch yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-cluster
namespace: elastic-system
spec:
version: 7.6.2
nodeSets:
- name: master
count: 3
config:
node.master: true
node.data: false
node.ingest: false
node.ml: false
node.store.allow_mmap: false
podTemplate:
...
- name: data1
count: 3
config:
node.master: false
node.data: true
node.ingest: true
node.ml: false
node.store.allow_mmap: false
index.refresh_interval: 120s # I added it here
podTemplate:
...
There is a third way, through kibana UI -> settings-> Elasticsearch -> Index management -> index template. But there is no index template for me to start with. Nevertheless, the Elasticsearch creates an index daily with the date. So, I do not want to mess with the existing template.
Can anyone suggest me a better way to do this. Thanks for your time.