Hello guys
I need to create an Index by manipulating another index with java.
here is my cluster specification:
ES version: 6.1
source Index documents count: 1.4 billion
the cluster had 2 nodes. both of them are data and master
each node: 50GB RAM, CPU 10* 2.2 GHz, SSD
process average speed ( read data from the source index by scrolling + manipulating + indexing in destination index): 250k per minute.
the problem is after about 1 day that was everything OK, I confront with search_context_missing_exception.
here is the log summary.
..
proceed : 620910000 / 1437095913
proceed : 620915000 / 1437095913
proceed : 620920000 / 1437095913
Exception in thread "main" ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]; nested: ElasticsearchException[Elasticsearch exception [type=search_context_missing_exception, reason=No search context found for id [235]]];
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:573)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:549)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:456)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:429)
at org.elasticsearch.client.RestHighLevelClient.searchScroll(RestHighLevelClient.java:387)
at Main.export(Main.java:201)
at Main.main(Main.java:71)
Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [https://el6.myserver.ir:443], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"search_context_missing_exception","reason":"No search context found for id [229]"},{"type":"search_context_missing_exception","reason":"No search context found for id [224]"},{"type":"search_context_missing_exception","reason":"No search context found for id [223]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":-1,"index":null,"reason":{"type":"search_context_missing_exception","reason":"No search context found for id [229]"}},{"shard":-1,"index":null,"reason":{"type":"search_context_missing_exception","reason":"No search context found for id [235]"}}],"caused_by":{"type":"search_context_missing_exception","reason":"No search context found for id [235]"}},"status":404}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:357)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:346)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
Caused by: ElasticsearchException[Elasticsearch exception [type=search_context_missing_exception, reason=No search context found for id [235]]]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:490)
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:406)
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:435)
at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:594)
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169)
... 7 more
Inserting : failed requests[5000], indices[m_web]ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [POST], host [https://el1.myserver.ir:443], URI [/_bulk?timeout=1m], status line [HTTP/1.1 504 Gateway Time-out]
and then I got a strange nginx error in the log :
The page is temporarily unavailablenginx error!
The page you are looking for is temporarily unavailable. Please try again later.
Website Administrator
Something has triggered an error on your website. This is the default error page for nginx that is distributed with Fedora. It is located /usr/share/nginx/html/50x.html
You should customize this error page for your own site or edit the error_page directive in the nginx configuration file /etc/nginx/nginx.conf.
and here is search request configuration:
SearchRequest searchRequest = new SearchRequest("myIndex");
searchRequest.types("myType");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
searchSourceBuilder.size(5000);
searchRequest.source(searchSourceBuilder);
sourceBuilder.timeout(new TimeValue(10, TimeUnit.MINUTES));
searchRequest.scroll(new TimeValue(10, TimeUnit.MINUTES));
I will be appreciated if someone help to resolve the problem.