Urgent - index templates lost after upgrade to 2.4.4 from 2.3.3

Index templates and repository settings lost after upgrade to 2.4.4 from 2.3.3(full cluster restart)
Is this expected behavior?
Where elasticsearch save index templates? what can I do is this situation?

any meta data store path configuration changed?

(all data, indicies are ok)

this is how I started elasticsearch 2.4.4

only one different configuration compared to es 2.3.3 configuration what I used is

-Des.bootstrap.memory_lock=true 
this changed id 2.4

full configuration for 2.4.4

myhome/apps/jdk/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Xms31g -Xmx31g -XX:-UseParNewGC -XX:-UseConcMarkSweepGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/myhome/.elasticsearch/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=1M 
-Des.path.home=/myhome/apps/elasticsearch -cp /myhome/apps/elasticsearch/lib/elasticsearch-2.4.4.jar:/myhome/apps/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d 
-Des.cluster.name=elastic_jpl 
-Des.node.name=xelastic101.band -Des.path.logs=/myhome/.elasticsearch/logs 
-Des.path.repo=/myhome/owfs/ba_elastic/snapshot 
-Des.discovery.zen.ping.unicast.hosts=... 
-Des.discovery.zen.minimum_master_nodes=2 
-Des.discovery.zen.ping_timeout=60s -Des.network.host=_eth1_ 
-Des.http.cors.allow-origin=/.*/ 
-Des.http.cors.allow-headers=X-Requested-With,Content-Type,Content-Length,content-type
-Des.http.cors.enabled=true 
-Des.node.max_local_storage_nodes=1 
-Des.bootstrap.memory_lock=true 
-Des.indices.fielddata.cache.size=30% 
-Des.indices.breaker.fielddata.limit=40% 
-Des.indices.breaker.request.limit=20% 
-Des.indices.store.throttle.max_bytes_per_sec=100mb -Des.node.master=false 
-Des.node.data=true -Des.path.data=/data1,/data2,/data3,/data4,/data5 
-Des.node.box_type=box_hot`

It should be in the cluster state. I can see that you are using multiple data path so may be there is a bug here? Did you change the data path or any other setting between 2.3 and 2.4 upgrade?

no, only memory lock config changed.
others are exactly same.
data directories are also fine.

where there templates saved?

It stores them in ES itself, ie as documents in a special location.

What were the steps you took to upgrade?

I have snapshot of some of indices and cluster global state.
Is is possible restore only global state and it will recover index templates?

I have upgraded as follow.

  1. shutdown all cluster gracefully.(2.3.3)

  2. replace elasticsearch directory

    unlink elasticsearch
    ln -sf elasticsearch-2.4.4 elasticsearch

all data, config and log directory are stored out of elasticsearch package directory.
3. started es 2.4.4 binary with almost same config.(as I described above)

I successfully recovered templates by restore snapshot.

Great!

That said this might indicate a bug when using multiple data path or something else. I'd open an issue.

1 Like

Thanks!
My lifetime first snapshot saved me. haha..

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.