Hello everyone. I've recently "inherited" elk stack at work. And there are problems with it. Kibana doesnt show any recent data, only logs that were recorded about week ago. In short there are 5 VMs in stack - 2 ES in cluster, 2 LS (load balance), and 1 Kibana.
It all started when 1 of LS nodes ran out of space. I've added some space to VM but after reboot all new logs stopped to aggregate. I've tried to restart ELK services on every node and each VM, but it doesnt help.
Currently there are errors at cluster logs:
[INFO ][o.e.i.s.GlobalCheckpointSyncAction] [elk-node-1] [php-2018.01.28][0] global checkpoint sync failed org.elasticsearch.transport.SendRequestTransportException: [elk-node-1][172.19.0.1:9300][indices:admin/seq_no/global_checkpoint_sync[p]]
and at one of LS node(both are suscepted):
[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[apache-2018.06.09][3] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[apache-2018.06.09][3]] containing [20] requests]"})
I'm completely confused due to lack of experience with ELK stack, so any advice will be much appreciated!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.