Courier Fetch: Failure ELK STACK

Elastic Stack Elasticsearch
1

Hi All,
I’m configure the ELK stack to analyze the syslog data. After Installing it worked for 4 months without any issues. Suddenly it starts to give the warning

Courier Fetch: 74 of 825 shards failed

Apart from that some times it gives the time out error on kibana dashboards.

I changed the resources configuration on elasticsearch.yml as follows.

01 thread_pool.bulk.queue_size: 200

02 threadpool.get.queue_size: 200

after doing change 01 problem not solved.

After performing change 02 elasticsearch process went down after 15 mins.

Here I attached the http://localhost:9200/_cat/shards?v output for reference.

Any help would be appreciated.
Thanks...

index shard prirep state docs store ip node

logstash-2017.09.17 2 p STARTED 2756 2.1mb 127.0.0.1 TJhnG3M
logstash-2017.09.17 2 r UNASSIGNED
logstash-2017.09.17 3 p STARTED 2800 2.2mb 127.0.0.1 TJhnG3M
logstash-2017.09.17 3 r UNASSIGNED
logstash-2017.09.17 1 p STARTED 2884 2.2mb 127.0.0.1 TJhnG3M
logstash-2017.09.17 1 r UNASSIGNED
logstash-2017.09.17 4 p STARTED 2999 2.2mb 127.0.0.1 TJhnG3M
logstash-2017.09.17 4 r UNASSIGNED
logstash-2017.09.17 0 p STARTED 2782 2.1mb 127.0.0.1 TJhnG3M
logstash-2017.09.17 0 r UNASSIGNED
logstash-2017.12.21 2 p STARTED 197281 119.4mb 127.0.0.1 TJhnG3M
logstash-2017.12.21 2 r UNASSIGNED
logstash-2017.12.21 1 p STARTED 197811 119.4mb 127.0.0.1 TJhnG3M
logstash-2017.12.21 1 r UNASSIGNED
logstash-2017.12.21 3 p STARTED 197782 120.5mb 127.0.0.1 TJhnG3M
logstash-2017.12.21 3 r UNASSIGNED
logstash-2017.12.21 4 p STARTED 196680 119mb 127.0.0.1 TJhnG3M
logstash-2017.12.21 4 r UNASSIGNED
logstash-2017.12.21 0 p STARTED 197098 118.1mb 127.0.0.1 TJhnG3M
logstash-2017.12.21 0 r UNASSIGNED
logstash-2018.02.27 2 p STARTED 37408 23.6mb 127.0.0.1 TJhnG3M
logstash-2018.02.27 2 r UNASSIGNED
logstash-2018.02.27 1 p STARTED 37324 23.4mb 127.0.0.1 TJhnG3M
logstash-2018.02.27 1 r UNASSIGNED
logstash-2018.02.27 3 p STARTED 37585 21.8mb 127.0.0.1 TJhnG3M
logstash-2018.02.27 3 r UNASSIGNED
logstash-2018.02.27 4 p STARTED 36928 21.4mb 127.0.0.1 TJhnG3M
logstash-2018.02.27 4 r UNASSIGNED
logstash-2018.02.27 0 p STARTED 36878 21.1mb 127.0.0.1 TJhnG3M
logstash-2018.02.27 0 r UNASSIGNED
logstash-2017.12.11 2 p STARTED 270760 168.1mb 127.0.0.1 TJhnG3M
logstash-2017.12.11 2 r UNASSIGNED
logstash-2017.12.11 1 p STARTED 271565 167mb 127.0.0.1 TJhnG3M
logstash-2017.12.11 1 r UNASSIGNED

logstash-2017.12.03 3 p STARTED 25313 10.9mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 3 r UNASSIGNED
logstash-2017.12.03 1 p STARTED 25337 11mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 1 r UNASSIGNED
logstash-2017.12.03 4 p STARTED 25567 11.1mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 4 r UNASSIGNED
logstash-2017.12.03 0 p STARTED 25002 10.9mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 0 r UNASSIGNED
logstash-2017.11.17 2 p STARTED 273766 162.4mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 2 r UNASSIGNED
logstash-2017.11.17 3 p STARTED 273367 163.5mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 3 r UNASSIGNED
logstash-2017.11.17 1 p STARTED 273632 164.1mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 1 r UNASSIGNED
logstash-2017.11.17 4 p STARTED 273793 163mb 127.0.0.1 TJhnG3M

logstash-2017.11.08 2 p STARTED 245968 139.7mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 2 r UNASSIGNED
logstash-2017.11.08 3 p STARTED 246007 139.9mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 3 r UNASSIGNED
logstash-2017.11.08 1 p STARTED 246933 140.8mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 1 r UNASSIGNED
logstash-2017.11.08 4 p STARTED 246712 140.4mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 4 r UNASSIGNED
logstash-2017.11.08 0 p STARTED 246922 140mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 0 r UNASSIGNED
logstash-2018.01.05 2 p STARTED 137960 91.5mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 2 r UNASSIGNED
logstash-2018.01.05 1 p STARTED 137454 91.1mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 1 r UNASSIGNED
logstash-2018.01.05 3 p STARTED 137453 90.9mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 3 r UNASSIGNED
logstash-2018.01.05 4 p STARTED 137905 91.2mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 4 r UNASSIGNED
logstash-2018.01.05 0 p STARTED 137419 90.9mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 0 r UNASSIGNED
logstash-2017.10.24 2 p STARTED 91806 67mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 2 r UNASSIGNED
logstash-2017.10.24 3 p STARTED 91519 66.9mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 3 r UNASSIGNED
logstash-2017.10.24 1 p STARTED 91292 66.7mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 1 r UNASSIGNED

2

It looks like you have a single node with a lot of small shards, which can be every inefficient and result in a lot of overhead. Read this blog post for some guidelines on how many shards you should aim to have in your cluster to get the most from it.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.