Script for safely re-opening an index - request for comments

Elastic Stack Elasticsearch
1

Hi

We have a constantly changing index that requires periodical re-opening in order to refresh synonym & other analyzers that depend on external data. Re-opening such an index is liable to turn the index red/yellow and a lot of rebalancing.

I've written the following Python script to try and and do this safely and quickly using the new flush synced and would love to hear your feedback. Caveats:

  • We use chef to deploy identical nodes and run scripts via cron but we want to only execute this once from the master.
  • We accept that ES will throw indexing errors while writes are disabled and will properly handle them at the client.

Script:

import time, logging, sys
from datetime import datetime
from elasticsearch import Elasticsearch
es = Elasticsearch()

if not es.nodes.info('_local')['nodes'].values()[0]['name'] == es.nodes.info('_master')['nodes'].values()[0]['name']:
	print "Reopen: Not master, aborting"
	sys.exit(0)

print "Reopen starting..."

if not es.cluster.health(wait_for_status='green', timeout=10)['status'] == 'green':
	print "Cluster status is not green, aborting"
	sys.exit(0)

try:
	print "Disabling writes..."
	es.indices.put_settings(index='our_index', body={"index.blocks.write": True})
	
	retries = 1
	while retries < 4:
	  print "Performing a synced flush, try #%s...." % retries
	  time.sleep(5)	

	  try:
	  	if es.indices.flush_synced('our_index').values()[0]['failed'] == 0:
	  		print "Synced flush successfull"
	  		break 
	  except:
	  	pass
	  	  
	  print "Synced flush failed, retrying..."
	  retries += 1

	if retries == 4:	  	
	  	raise Exception("Too many retries, aborting")

	print "Closing..."
	es.indices.close('our_index')

	print "Opening..."
	es.indices.open('our_index')

	time.sleep(2)


except Exception as e:
	logging.exception(e)

finally:
	print "Enalbing writes..."
	es.indices.put_settings(index='our_index', body={"index.blocks.write": False})

print "Reopen done..."

Thanks,
Barak