Elasticsearch: TransportError exception while searching Id

(Nilesh Gaikwad) #1

Hello All,

I'm new to Elasticsearch, while running below python code I'm getting TransportError: TransportError(503, u'search_phase_execution_exception') :-

Python code :-

	f = open(file, "rb")
	strings = f.read(offset)
	strings =  binascii.hexlify(strings)
	strings = ' '.join(strings[i:i+2] for i in xrange(0,len(strings),2))
	strings = strings + " "
	string = strings.split(" 00 ")

	for stri in string:
		if stri != "":
			if stri == "00":
			temp = stri.replace(" ","") + "00"

			occurence = 1
				**res =  es_client.search( index=index, doc_type=type, body={"query": { "match":{"_id":temp} } })**
				if res['hits']['hits']:
					for src_dict in res['hits']['hits']:
						if src_dict['_source']['occurence']:
							occurence = src_dict['_source']['occurence'] + 1
				print traceback.print_exc()

			json_data = {}
			json_data['_index'] = index
			json_data['_type'] = type
			json_data['_id'] = temp
			json_data['_source'] = { 'occurence' : occurence,
								'timestamp' : datetime.now()
			del json_data

				if len(bulk_res_list) >= BULK_MAX:
					helpers.bulk(es_client, bulk_res_list)
					del bulk_res_list[:]
				print traceback.print_exc()
		if len(bulk_res_list) >= 0:
			helpers.bulk(es_client, bulk_res_list)
			del bulk_res_list[:]
		print traceback.print_exc()


I'm getting below exceptions :-

	Traceback (most recent call last):
	  File "updateStringOccurence.py", line 443, in insertFunction
		**res =  es_client.search( index=index, doc_type=type, body={"query": { "match":{"_id":temp} } })**
	  File "C:\Python27\lib\site-packages\elasticsearch\client\utils.py", line 69, in _wrapped
		return func(*args, params=params, **kwargs)
	  File "C:\Python27\lib\site-packages\elasticsearch\client\__init__.py", line 530, in search
		doc_type, '_search'), params=params, body=body)
	  File "C:\Python27\lib\site-packages\elasticsearch\transport.py", line 307, in perform_request
		status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
	  File "C:\Python27\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 93, in perform_request
		self._raise_error(response.status, raw_data)
	  File "C:\Python27\lib\site-packages\elasticsearch\connection\base.py", line 105, in _raise_error
		raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
	**TransportError: TransportError(503, u'search_phase_execution_exception')**

Below are the node details :-

O.S :- CentOS release 6.7 (Final)
ES version:- 2.2.2

(Mark Walkom) #2

Have a look in your ES logs as well, there should be something at around the same time.

(Nilesh Gaikwad) #3

Hello Mark, Thanks for your reply.

I'm getting below Warnings and exceptions inside ES logs.

[2016-05-05 02:19:39,551][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed

[2016-05-05 02:19:40,192][WARN ][env                      ] [node189] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536]

[2016-05-05 02:20:03,207][DEBUG][action.search.type       ] [node189] All shards failed for phase: [query]
RemoteTransportException[[node189][][indices:data/read/search[phase/query]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [stringoccurences][[stringoccurences][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
	at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1037)
	at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:851)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:641)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:618)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:369)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

[2016-05-05 02:20:03,210][WARN ][rest.suppressed          ] /stringoccurences/strings/_search Params: {index=stringoccurences, type=strings}
Failed to execute phase [query], all shards failed
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228)
	at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:174)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
	at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

I have tried to solve the "All shards failed for phase:" exception previously with following command.

for shard in $(curl -XGET | grep UNASSIGNED | awk '{print $2}')
curl -XPOST -d '{
"commands" : [ {
"allocate" : {
"index" : "stringoccurences",
"shard" : '$shard',
"node" : "node189",
"allow_primary" : true

I'm getting below output(may be because I'm using single node ):-

	[NO(shard cann ot be allocated on same node [f-7x-7zaSgaT7OUiHV66EQ] it already exists on)] 

I also don't understand why ES using port "9300":- RemoteTransportException[[node189][]
even after I have specified port as "9200" inside .yml file

(Nilesh Gaikwad) #4

Hello All,

I have solved the above error by myself :stuck_out_tongue: , sharing it here so it will be useful for others,
following are the points I have added/modified:-

  1. Previously I was not checking if the index is already present or not, so I have added the code to check for it. And I was searching inside the index which is newly created, which was the one reason I was getting "TransportError".

  2. For that, I have added some sleep time after the creation of new index if it is not present. I do not know the actual reason for sleep(60 seconds) working, but if I do not use sleep after creating new index I'm getting "TransportError".

  3. I'm using get API after checking if the document exist using exists API, instead of search API.

  4. I have added below attributes in elasticsearch.yml file;

     index.refresh_interval: 1s
     index.number_of_replicas: 0
     index.translog.flush_threshold_ops: 50000
     index.translog.flush_threshold_size: 1gb
     indices.store.throttle.max_bytes_per_sec: 20mb
     threadpool.bulk.type: fixed
     threadpool.bulk.size: 12
     threadpool.bulk.queue_size: 200000
     index.merge.scheduler.max_thread_count: 1

any suggestions are welcome :slight_smile:


(vitthal ude) #5

Hi Nilesh,

Could you please provide the code for the steps you performed to resolve the issue?

Thanks alot in advance

(Nilesh Gaikwad) #6

Hello Vitthal,

I have already added steps in my previous reply.. Those are only steps that I had performed..


(system) #7