Templates don't persist after service restart

Elastic Stack Elasticsearch
1

So I'm trying to set templates while elasticsearch is not available to the rest of the network, and then restart elasticsearch in a mode that is available to the rest of the network. When elasticsearch is restarted, the templates are gone which defeats the purpose. What is happening here?

The reason I am doing this, is because I want to ensure that everything in Elasticsearch is set to 1 shard and 0 replicas. If I do not restrict network access to Elasticsearch, Filebeat and Kibana will keep trying to contact Elasticsearch and sometimes will jump in and create an index that isn't 1 shard and 0 replicas before my templates get applied. So this is my solution right now. If anyone has any ideas I'd love to hear them. Using iptables is not an option.

Elasticsearch 2.3.3 - upgrading is not an option at the moment.

Bash script that inits everything:

set -m

su -c "/opt/elasticsearch/bin/elasticsearch -Des.default.path.home=/opt/elasticsearch \
  -Des.default.path.logs=/opt/elasticsearch/logs \
  -Des.default.path.data=/opt/elasticsearch/data \
  -Des.default.path.work=/opt/elasticsearch/tmp \
  -Des.discovery.zen.ping.multicast.enabled=false \
  -Des.threadpool.bulk.queue_size=500 \
  -Des.node.name=localNode \
  -Des.network.host=_local_" elasticsearch &

curl -XPUT 'http://elasticsearch:9200/_template/global_1' -H 'Content-Type: application/json' -d'
{
	"template": "*",
	"order": 9,
	"settings": {
		"number_of_shards": 1,
		"number_of_replicas": 0
	}
}
'
curl -XPUT 'http://elasticsearch:9200/_template/kibana_1' -H 'Content-Type: application/json' -d'
{
	"template": ".kibana",
	"order": 10,
	"settings": {
		"number_of_shards": 1,
		"number_of_replicas": 0
	}
}
'
kill $!

echo "Elasticsearch Init Complete - switching Elasticsearch to foreground and opening up to the world"
su -c "/opt/elasticsearch/bin/elasticsearch -Des.default.path.home=/opt/elasticsearch \
  -Des.default.path.logs=/opt/elasticsearch/logs \
  -Des.default.path.data=/opt/elasticsearch/data \
  -Des.default.path.work=/opt/elasticsearch/tmp \
  -Des.discovery.zen.ping.multicast.enabled=false \
  -Des.threadpool.bulk.queue_size=500 \
  -Des.node.name=localNode \
  -Des.network.host=_local_,_site_" elasticsearch
2

Are you saying that when you do the above and then restart, and check _template these don't exist?

3

Well, the above does an elasticsearch restart (stops the first elasticsearch that only has _local_ and then starts one with _local_ and _site_).

But yes, checking _template after restart yields nothing. Checking _template after the -XPUT commands does show the templates though, but the restart makes everything disappear.

4

Ok, got this working by just adding the replica and shard settings to the JAVA options!

su -c "/opt/elasticsearch/bin/elasticsearch -Des.default.path.home=/opt/elasticsearch \
  -Des.default.path.logs=/opt/elasticsearch/logs \
  -Des.default.path.data=/opt/elasticsearch/data \
  -Des.default.path.work=/opt/elasticsearch/tmp \
  -Des.discovery.zen.ping.multicast.enabled=false \
  -Des.threadpool.bulk.queue_size=500 \
  -Des.node.name=localNode \
  -Des.network.host=_local_,_site_ \
  -Des.index.number_of_replicas=0 \
  -Des.index.number_of_shards=1" elasticsearch
5

Configuring index settings on the command line is not a solution: this ability was removed in 5.x. Some things I notice with the command line that should be fixed to ensure it is not something silly with how paths are handled:

  • There should be no need to set default path values, just override the path values. In fact, no need to set any path values except for path.home (logs and data will be relative to that by default)
  • path.work is not a valid setting
  • orthogonally: the multicast setting should not be necessary on 2.x as it was moved to a plugin in 2.0
1 Like
6

I would take a look under /opt/elasticsearch/data/nodes and see if something is causing the second instance to come up with a different node number.

7

Finally got it working, thank you so much @rjernst and @Badger! Problem was actually just that I wasn't waiting long enough for Elasticsearch to start. At first I got this working with a sleep, but then refined it and now I actually query the health check endpoint to see when we are ready to set stuff.

Final working init:

set -m

su -c "/opt/elasticsearch/bin/elasticsearch -Des.path.home=/opt/elasticsearch \
  -Des.threadpool.bulk.queue_size=500 \
  -Des.node.name=localNode \
  -Des.network.host=_local_" elasticsearch &

echo "Waiting for Elasticsearch to start"
while true; do
  HEALTH=$(curl http://localhost:9200/_cluster/health?wait_for_status=yellow -s)
  RETVAL1=$?
  HEALTH2=$(curl http://localhost:9200/_cluster/health?wait_for_status=green -s)
  RETVAL2=$?
  if [ ${RETVAL1:-25} -eq 0 ] || [ ${RETVAL2:-25} -eq 0  ]; then
echo "Elasticsearch is ready. Proceeding"
break
  fi
done


curl -XPUT 'http://localhost:9200/_template/global_1' -H 'Content-Type: application/json' -d'
{
	"template": "*",
	"order": 9,
	"settings": {
		"number_of_shards": 1,
		"number_of_replicas": 0
	}
}
'
curl -XPUT 'http://localhost:9200/_template/kibana_1' -H 'Content-Type: application/json' -d'
{
	"template": ".kibana",
	"order": 10,
	"settings": {
		"number_of_shards": 1,
		"number_of_replicas": 0
	}
}
'
kill $!

echo "Elasticsearch Settings Apply Complete - switching Elasticsearch to foreground and opening up to the world"
su -c "/opt/elasticsearch/bin/elasticsearch -Des.path.home=/opt/elasticsearch \
  -Des.threadpool.bulk.queue_size=500 \
  -Des.node.name=localNode \
  -Des.network.host=_local_,_site_" elasticsearch
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.