Urgent - index templates lost after upgrade to 2.4.4 from 2.3.3

[no_jihun]

Index templates and repository settings lost after upgrade to 2.4.4 from 2.3.3(full cluster restart)
Is this expected behavior?
Where elasticsearch save index templates? what can I do is this situation?

any meta data store path configuration changed?

(all data, indicies are ok)

this is how I started elasticsearch 2.4.4

only one different configuration compared to es 2.3.3 configuration what I used is

-Des.bootstrap.memory_lock=true 
this changed id 2.4

full configuration for 2.4.4

myhome/apps/jdk/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Xms31g -Xmx31g -XX:-UseParNewGC -XX:-UseConcMarkSweepGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/myhome/.elasticsearch/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=1M 
-Des.path.home=/myhome/apps/elasticsearch -cp /myhome/apps/elasticsearch/lib/elasticsearch-2.4.4.jar:/myhome/apps/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d 
-Des.cluster.name=elastic_jpl 
-Des.node.name=xelastic101.band -Des.path.logs=/myhome/.elasticsearch/logs 
-Des.path.repo=/myhome/owfs/ba_elastic/snapshot 
-Des.discovery.zen.ping.unicast.hosts=... 
-Des.discovery.zen.minimum_master_nodes=2 
-Des.discovery.zen.ping_timeout=60s -Des.network.host=_eth1_ 
-Des.http.cors.allow-origin=/.*/ 
-Des.http.cors.allow-headers=X-Requested-With,Content-Type,Content-Length,content-type
-Des.http.cors.enabled=true 
-Des.node.max_local_storage_nodes=1 
-Des.bootstrap.memory_lock=true 
-Des.indices.fielddata.cache.size=30% 
-Des.indices.breaker.fielddata.limit=40% 
-Des.indices.breaker.request.limit=20% 
-Des.indices.store.throttle.max_bytes_per_sec=100mb -Des.node.master=false 
-Des.node.data=true -Des.path.data=/data1,/data2,/data3,/data4,/data5 
-Des.node.box_type=box_hot`

[dadoonet (David Pilato)]

It should be in the cluster state. I can see that you are using multiple data path so may be there is a bug here? Did you change the data path or any other setting between 2.3 and 2.4 upgrade?

[no_jihun]

no, only memory lock config changed.
others are exactly same.
data directories are also fine.

where there templates saved?

[warkolm (Mark Walkom)]

It stores them in ES itself, ie as documents in a special location.

What were the steps you took to upgrade?

[no_jihun]

I have snapshot of some of indices and cluster global state.
Is is possible restore only global state and it will recover index templates?

[no_jihun]

I have upgraded as follow.

1. shutdown all cluster gracefully.(2.3.3)

2. replace elasticsearch directory

   unlink elasticsearch
   ln -sf elasticsearch-2.4.4 elasticsearch

all data, config and log directory are stored out of elasticsearch package directory.
3. started es 2.4.4 binary with almost same config.(as I described above)

[no_jihun]

I successfully recovered templates by restore snapshot.

[dadoonet (David Pilato)]

Great!

That said this might indicate a bug when using multiple data path or something else. I'd open an issue.

[no_jihun]

Thanks!
My lifetime first snapshot saved me. haha..