Hello ,
When I access to dashbord in Kibana I have this message in elasticsearch logs .
did you encouter such problems ?
thanks
Sc2w4wt3q-7cZg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[*], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=18, source={"size":0,"query":{"bool":{"must":[{"range":{"@timestamp":{"from":1517780215222,"to":1517823415222,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"bool":{"must":[{"query_string":{"query":"beat.name:"vm-mule-25"","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"993acf30-1b19-11e7-b09e-037021c4f8df":{"filter":{"match_all":{"boost":1.0}},"aggregations":{"timeseries":{"date_histogram":{"field":"@timestamp","time_zone":"Europe/Berlin","interval":"300s","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":1517780215222,"max":1517823415222}},"aggregations":{"993acf31-1b19-11e7-b09e-037021c4f8df":{"avg":{"field":"system.cpu.system.pct"}}}}}}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [elacticsearch.unealgroupe.fr][192.168.9.160:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@2e110ea4 on QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 721nanos, adjustment amount = 50, QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@4420358a[Running, pool size = 4, active threads = 4, queued tasks = 1084, completed tasks = 3885]]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.1.1.jar:6.1.1]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_131]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:93) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$0(SearchService.java:988) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:113) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:86) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.search.SearchService.rewriteShardRequest(SearchService.java:986) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:376) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:373) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:646) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:135) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:594) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:514) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:554) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:545) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:156) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:52) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.InitialSearchPhase.performPhaseOnShard(InitialSearchPhase.java:208) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.InitialSearchPhase.run(InitialSearchPhase.java:153) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:146) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:116) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.TransportSearchAction$1.run(TransportSearchAction.java:393) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:146) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:140) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.1.1.jar:6.1.1]
....
Your node is overloaded and is rejecting request to make sure it can stay afloat. You should add more nodes, more resources and/or both/
How many data nodes do you have in your cluster ?
How many indices/size are your backing up each day ?
It is a one node for testing .
how can I check the number of indexes .
Please could you suggest me a documentation for production architecture
thanks you
In kibana I get this message
Visualize: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[],"caused_by":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@77b0e659 on QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 66.5micros, adjustment amount = 50, QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@4420358a[Running, pool size = 4, active threads = 4, queued tasks = 1012, completed tasks = 108762]]]"}},"status":503}
less
You can find all the information you need here: https://www.elastic.co/guide/index.html
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.