KIBANA - Elastic search PLUGIN IS RED, UI SETTINGS PLUGN IS RED

Elastic Stack Kibana
1

image
image.png2063×931 85.1 KB

LOGS FROM data ingestion server
failed to execute pipeline for a bulk request
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.ingest.PipelineExecu
tionService$2@471e0de7 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExe
cutor@73250f05[Running, pool size = 2, active threads = 2, queued tasks = 205, completed tasks = 12360]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.5.2.ja

LOGS FROM ELASTIC MASTER
[r.suppressed ] path: /_template/filebeat-6.1.2, params: {name=filebeat-6.1.2}
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (create-index-template [fil
ebeat-6.1.2], cause [api]) within 30s

2

What is the status of your Elasticsearch cluster? Is there anything in the logs?

3

Status of ES cluster is running(master, data nodes are running). I have attached the corresponding logs above post

4

What is the specification of your cluster? How many indices and shards do you have in it?

5

I was able to get some far on the previous issue Now I have a different one
KIBANA LOGS kibana.stdout.log
{"type":"response","@timestamp":"2019-09-13T15:43:03Z","tags":,"pid":6528,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch","method":"post","headers":{"host":"logs","content-length":"865","accept":"application/json, text/plain, /","origin":"https://logs","kbn-version":"5.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","sec-fetch-mode":"cors","content-type":"application/x-ndjson","sec-fetch-site":"same-origin","referer":"https://logs/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.249.0.7","userAgent":"10.249.0.7","referer":"https://logs/app/kibana"},"res":{"statusCode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/_msearch 200 32ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-13T15:43:28Z","tags":["error","elasticsearch","admin"],"pid":6528,"message":"Request error, retrying\nPOST http://127.0.0.1:9200/.kibana/config/5.5.2/_update => socket hang up"}
{"type":"log","@timestamp":"2019-09-13T15:43:58Z","tags":["error","elasticsearch","admin"],"pid":6528,"message":"Request error, retrying\nPOST http://127.0.0.1:9200/.kibana/config/5.5.2/_update => socket hang up"}

6

1 master node
3 data nodes
1 haproxy
1 monitor for cluster

7

I have the following server configuriation (master, 3 clusters,1haproxy)aster is reporting this
Elastic Search Master Logs
[elasticsearch_data/0][1.X.X.X:9300][indices:data/write/bulk]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.5.2.jar:5
.5.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:396) ~[el
asticsearch-5.5.2.jar:5.5.2]7

Elastic Search data node LOGS
[o.e.d.z.ZenDiscovery ] [elasticsearch_data/0] not enough master nodes discovered during pingin
g (found [], but needed [1]), pinging again

KIBANA LOGS
{"type":"response","@timestamp":"2019-09-13T15:43:03Z","tags":,"pid":6528,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch","method":"post","headers":{"host":"logs","content-length":"865","accept":"application/json, text/plain, / ","origin":"https://logs","kbn-version":"5.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","sec-fetch-mode":"cors","content-type":"application/x-ndjson","sec-fetch-site":"same-origin","referer":"https://logs/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.249.0.7","userAgent":"10.249.0.7","referer":"https://logs/app/kibana"},"res":{"statusCode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/_msearch 200 32ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-13T15:43:28Z","tags":["error","elasticsearch","admin"],"pid":6528,"message":"Request error, retrying\nPOST http://127.0.0.1:9200/.kibana/config/5.5.2/_update => socket hang up"}
{"type":"log","@timestamp":"2019-09-13T15:43:58Z","tags":["error","elasticsearch","admin"],"pid":6528,"message":"Request error, retrying\nPOST http://127.0.0.1:9200/.kibana/config/5.5.2/_update => socket hang up"}**

CLUSTER HEALTH DOESNT RESPOND
curl http://10.X.X.X:9200/_cluster/health
no reponse

8

Is your Elasticsearch cluster running?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.