Elasticsearch: TransportError exception while searching Id

Hello All,

I'm new to Elasticsearch, while running below python code I'm getting TransportError: TransportError(503, u'search_phase_execution_exception') :-

Python code :-

	f = open(file, "rb")
	strings = f.read(offset)
	strings =  binascii.hexlify(strings)
	strings = ' '.join(strings[i:i+2] for i in xrange(0,len(strings),2))
	strings = strings + " "
	string = strings.split(" 00 ")

	for stri in string:
		if stri != "":
			if stri == "00":
				continue
			temp = stri.replace(" ","") + "00"

			occurence = 1
			try:
				**res =  es_client.search( index=index, doc_type=type, body={"query": { "match":{"_id":temp} } })**
				if res['hits']['hits']:
					for src_dict in res['hits']['hits']:
						if src_dict['_source']['occurence']:
							occurence = src_dict['_source']['occurence'] + 1
			except:
				print traceback.print_exc()

			json_data = {}
			json_data['_index'] = index
			json_data['_type'] = type
			json_data['_id'] = temp
			json_data['_source'] = { 'occurence' : occurence,
								'timestamp' : datetime.now()
						}
			bulk_res_list.append(json_data)
			del json_data

			try:
				if len(bulk_res_list) >= BULK_MAX:
					helpers.bulk(es_client, bulk_res_list)
					del bulk_res_list[:]
			except:
				print traceback.print_exc()
	try:
		if len(bulk_res_list) >= 0:
			helpers.bulk(es_client, bulk_res_list)
			del bulk_res_list[:]
	except:
		print traceback.print_exc()

	f.close()

I'm getting below exceptions :-

	Traceback (most recent call last):
	  File "updateStringOccurence.py", line 443, in insertFunction
		**res =  es_client.search( index=index, doc_type=type, body={"query": { "match":{"_id":temp} } })**
	  File "C:\Python27\lib\site-packages\elasticsearch\client\utils.py", line 69, in _wrapped
		return func(*args, params=params, **kwargs)
	  File "C:\Python27\lib\site-packages\elasticsearch\client\__init__.py", line 530, in search
		doc_type, '_search'), params=params, body=body)
	  File "C:\Python27\lib\site-packages\elasticsearch\transport.py", line 307, in perform_request
		status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
	  File "C:\Python27\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 93, in perform_request
		self._raise_error(response.status, raw_data)
	  File "C:\Python27\lib\site-packages\elasticsearch\connection\base.py", line 105, in _raise_error
		raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
	**TransportError: TransportError(503, u'search_phase_execution_exception')**

Below are the node details :-

O.S :- CentOS release 6.7 (Final)
ES version:- 2.2.2

Have a look in your ES logs as well, there should be something at around the same time.

Hello Mark, Thanks for your reply.

I'm getting below Warnings and exceptions inside ES logs.

[2016-05-05 02:19:39,551][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-05-05 02:19:40,192][WARN ][env                      ] [node189] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536]

[2016-05-05 02:20:03,207][DEBUG][action.search.type       ] [node189] All shards failed for phase: [query]
RemoteTransportException[[node189][10.10.97.42:9300][indices:data/read/search[phase/query]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [stringoccurences][[stringoccurences][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
	at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1037)
	at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:851)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:641)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:618)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

[2016-05-05 02:20:03,210][WARN ][rest.suppressed          ] /stringoccurences/strings/_search Params: {index=stringoccurences, type=strings}
Failed to execute phase [query], all shards failed
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228)
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:174)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
	at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

I have tried to solve the "All shards failed for phase:" exception previously with following command.

for shard in $(curl -XGET http://10.10.97.42:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}')
do
curl -XPOST 10.10.97.42:9200/_cluster/reroute -d '{
"commands" : [ {
"allocate" : {
"index" : "stringoccurences",
"shard" : '$shard',
"node" : "node189",
"allow_primary" : true
}
}
]
}'
done

I'm getting below output(may be because I'm using single node ):-

	[NO(shard cann ot be allocated on same node [f-7x-7zaSgaT7OUiHV66EQ] it already exists on)] 

I also don't understand why ES using port "9300":- RemoteTransportException[[node189][10.10.97.42:9300]
even after I have specified port as "9200" inside .yml file

Hello All,

I have solved the above error by myself :stuck_out_tongue: , sharing it here so it will be useful for others,
following are the points I have added/modified:-

  1. Previously I was not checking if the index is already present or not, so I have added the code to check for it. And I was searching inside the index which is newly created, which was the one reason I was getting "TransportError".

  2. For that, I have added some sleep time after the creation of new index if it is not present. I do not know the actual reason for sleep(60 seconds) working, but if I do not use sleep after creating new index I'm getting "TransportError".

  3. I'm using get API after checking if the document exist using exists API, instead of search API.

  4. I have added below attributes in elasticsearch.yml file;

     index.refresh_interval: 1s
     index.number_of_replicas: 0
     index.translog.flush_threshold_ops: 50000
     index.translog.flush_threshold_size: 1gb
     indices.store.throttle.max_bytes_per_sec: 20mb
    
     threadpool.bulk.type: fixed
     threadpool.bulk.size: 12
     threadpool.bulk.queue_size: 200000
    
     index.merge.scheduler.max_thread_count: 1
    

any suggestions are welcome :slight_smile:

Thanks.

Hi Nilesh,

Could you please provide the code for the steps you performed to resolve the issue?

Thanks alot in advance

Hello Vitthal,

I have already added steps in my previous reply.. Those are only steps that I had performed..

Thanks.