Elasticsearch: Apply index templates at start-up

Elastic Stack Elasticsearch
1

Hi,

I was wondering if it's possible to start up Elasticsearch with index templates applied (for example by being defined in a config).

We currently have a pod that has Elastic and Logstash containers, and the Logstash pipeline applies the index templates at startup.

But, sometimes, for some reason, the ES container in the pod restarts. Logstash doesn't re-apply the template (since Logstash keeps running), pushes updates (and thus creating the index with an invalid mapping) and we would get errors on our backend that certain nested fields aren't present (that are defined in the template).

We would like to move to a solution where ES itself applies the index template before being available, for robustness reasons.

We thought about applying the templates in the ES container itself at startup by calling the relevant REST endpoints, but then Logstash would still be able to push data, possibly when the index template hasn't been applied yet. Setting a readiness probe on Kubernetes level doesn't matter since both containers are running in the same pod.

Is this possible? Are there any alternative solutions?

Thanks!

2

You may need to add some more context here. Index templates are stored in the cluster state and are loaded when a node starts up again. In your case it sounds, as if the elasticsearch node is a completely new node, forming its own new cluster, that does not have any historical data or configuration.

I don't think that this is the wanted setup. Maybe you can elaborate?

3

We aren't running Elasticsearch in clustered mode. We are running it in a Docker container in a Kubernetes pod. We have our Elasticsearch container configured with discovery.type: single-node.

If there is a cluster/node state, we aren't persisting it. The ES container is stateless, at the moment. We expect Logstash to re-persist all data in case the pod goes down.

4

I'm wondering if we can solve this issue in a better way, by perhaps persisting SOME state of the Elastic container?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.