Hello,
Do you know what might be the reason of this error? It's very annoying
{"type":"log","@timestamp":"2019-09-08T14:50:12Z","tags":["error","task_manager"],"pid":1,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2019-09-08T14:50:45Z","tags":["error","task_manager"],"pid":1,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"response","@timestamp":"2019-09-08T14:50:09Z","tags":[],"pid":1,"method":"post","statusCode":503,"req":{"url":"/api/security/v1/login","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","kbn-version":"7.2.0","content-length":"44","connection":"keep-alive","referer":"http://localhost:5601/login?next=%2F"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/login?next=%2F"},"res":{"statusCode":503,"responseTime":35757,"contentLength":9},"message":"POST /api/security/v1/login 503 35757ms - 9.0B"}
Elasticsearch master0 logs:
{"type": "server", "timestamp": "2019-09-08T14:28:15,092+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "6lJ-oAsCQYi5Tndu0CpWDw", "node.id": "FsyZhvdITuqkUSYbgMSqRA", "message": "Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...])." }
Elasticsearch master1 logs:
{"type": "server", "timestamp": "2019-09-08T14:23:59,035+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "6lJ-oAsCQYi5Tndu0CpWDw", "node.id": "E9Dj0FZmT22bVUZsWOwHLw", "message": "All shards failed for phase: [query]" }
{"type": "server", "timestamp": "2019-09-08T14:23:59,036+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "6lJ-oAsCQYi5Tndu0CpWDw", "node.id": "E9Dj0FZmT22bVUZsWOwHLw", "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}" ,
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.2.0.jar:7.2.0]",
curl -u elastic http://localhost:9200/_cluster/health?pretty
Enter host password for user 'elastic':
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 3,
"active_shards" : 6,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
I haven't started any log collection tool like fluentd yet
Update: restarting por with error all shards failed" solved login issue.
Do you know what is the root cause?
