Elastic Cluster connection Issue

Hi,
We have frequently faced the below issue
" Can not connect to the Elastic Search Cluster
See the Kibana logs for details and try to reload the page"
Someone pls help me on this!!

I do not think anyone can help with that little information. Please provide the following information:

  • Which Elasticsearch version are you using?
  • What is the size and specification of your cluster?
  • How is the cluster configured?
  • What is in the Elasticsearch logs?
  • What is in the Kibana logs?
1 Like

Welcome to our community! :smiley:

Please take a look at Dec 10th, 2022: [EN] Asking top notch technical questions to get you help quicker! and then update your post with more information to allow us to help you.

Hi Chris,
version - 7.1.1
VM - Standard F8s v2 (8 vcpus, 16 GiB memory) - Azure.

Elastic Search Logs:
[2023-04-04T00:03:02,201][INFO ][o.e.c.m.MetaDataCreateIndexService] [ELK-01] [filebeat-2023.14.04] creating index, cause [auto(bulk api)], templates , shards [1]/[1], mappings
[2023-04-04T00:03:02,967][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] create_mapping [_doc]
[2023-04-04T00:03:02,967][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:03,029][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:05,908][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:05,986][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:05,986][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:06,049][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:03:06,174][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:04:30,312][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T00:08:14,932][WARN ][o.e.m.j.JvmGcMonitorService] [ELK-01] [gc][14009] overhead, spent [5.1s] collecting in the last [5.6s]
[2023-04-04T00:08:15,858][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ELK-01] adding template [.management-beats] for index patterns [.management-beats]
[2023-04-04T00:24:10,346][INFO ][o.e.c.m.MetaDataMappingService] [ELK-01] [filebeat-2023.14.04/kF_8fr2CSfO8DtPUPDux5Q] update_mapping [_doc]
[2023-04-04T13:22:20,985][INFO ][o.e.m.j.JvmGcMonitorService] [ELK-01] [gc][old][61268][6820] duration [5s], collections [1]/[5.5s], total [5s]/[8.5m], memory [3.6gb]->[3.3gb]/[3.9gb], all_pools {[young] [229.8mb]->[5mb]/[532.5mb]}{[survivor] [42.7mb]->[0b]/[66.5mb]}{[old] [3.3gb]->[3.3gb]/[3.3gb]}
[2023-04-04T13:22:20,985][WARN ][o.e.m.j.JvmGcMonitorService] [ELK-01] [gc][61268] overhead, spent [5s] collecting in the last [5.5s]
[2023-04-04T14:18:32,762][DEBUG][o.e.a.s.TransportSearchAction] [ELK-01] [filebeat-2022.26.27][0], node[gVBS2KYESYyXpCKoxv7PYw], [P], s[STARTED], a[id=ZDK5r21YRvy0wDNXopAMMw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[filebeat-], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"timeout":"1s","terminate_after":100000,"query":{"match_all":{"boost":1.0}},"aggregations":{"suggestions":{"terms":{"field":"Message.keyword","size":10,"shard_size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"execution_hint":"map","order":[{"_count":"desc"},{"_key":"asc"}],"include":"."}}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [ELK-01][199.50.0.29:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:307) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$18(IndicesService.java:1272) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$19(IndicesService.java:1329) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:123) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1335) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1269) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:347) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:393) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.access$100(SearchService.java:124) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:358) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1069) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<reused_arrays>] would be [4014880416/3.7gb], which is larger than the limit of [4013975142/3.7gb], real usage: [4014863976/3.7gb], new bytes reserved: [16440/16kb]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:343) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.BigArrays.adjustBreaker(BigArrays.java:399) ~[elasticsearch-7.1.1.jar:7.1.1],

I have faced this error repeatedly in my Kibana Visualization:

This is very, very old and has been EOL a long time. I would recommend you upgrade at least to version 7.17 as soon as possible.

You are having problems with high heap usage and slow GC. What is the full output of the cluster stats API?

Also please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.