Courier Fetch: Failure ELK STACK

Methsri_Wickramarath · February 28, 2018, 5:04am

Hi All,
I’m configure the ELK stack to analyze the syslog data. After Installing it worked for 4 months without any issues. Suddenly it starts to give the warning

Courier Fetch: 74 of 825 shards failed

Apart from that some times it gives the time out error on kibana dashboards.

I changed the resources configuration on elasticsearch.yml as follows.

01 thread_pool.bulk.queue_size: 200

02 threadpool.get.queue_size: 200

after doing change 01 problem not solved.

After performing change 02 elasticsearch process went down after 15 mins.

Here I attached the http://localhost:9200/_cat/shards?v output for reference.

Any help would be appreciated.
Thanks...

index shard prirep state docs store ip node

logstash-2017.09.17 2 p logstash-2017.09.17 2 r logstash-2017.09.17 3 p logstash-2017.09.17 3 r logstash-2017.09.17 1 p logstash-2017.09.17 1 r logstash-2017.09.17 4 p logstash-2017.09.17 4 r logstash-2017.09.17 0 p logstash-2017.09.17 0 r logstash-2017.12.21 2 p logstash-2017.12.21 2 r logstash-2017.12.21 1 p logstash-2017.12.21 1 r logstash-2017.12.21 3 p logstash-2017.12.21 3 r logstash-2017.12.21 4 p logstash-2017.12.21 4 r logstash-2017.12.21 0 p logstash-2017.12.21 0 r logstash-2018.02.27 2 p logstash-2018.02.27 2 r logstash-2018.02.27 1 p logstash-2018.02.27 1 r logstash-2018.02.27 3 p logstash-2018.02.27 3 r logstash-2018.02.27 4 p logstash-2018.02.27 4 r logstash-2018.02.27 0 p logstash-2018.02.27 0 r logstash-2017.12.11 2 p logstash-2017.12.11 2 r logstash-2017.12.11 1 p logstash-2017.12.11 1 r STARTED 2756 2.1mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 2800 2.2mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 2884 2.2mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 2999 2.2mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 2782 2.1mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 197281 119.4mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 197811 119.4mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 197782 120.5mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 196680 119mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 197098 118.1mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 37408 23.6mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 37324 23.4mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 37585 21.8mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 36928 21.4mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 36878 21.1mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 270760 168.1mb 127.0.0.1 TJhnG3M
UNASSIGNED
STARTED 271565 167mb 127.0.0.1 TJhnG3M
UNASSIGNED

logstash-2017.12.03 3 p STARTED 25313 10.9mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 3 r UNASSIGNED
logstash-2017.12.03 1 p STARTED 25337 11mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 1 r UNASSIGNED
logstash-2017.12.03 4 p STARTED 25567 11.1mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 4 r UNASSIGNED
logstash-2017.12.03 0 p STARTED 25002 10.9mb 127.0.0.1 TJhnG3M
logstash-2017.12.03 0 r UNASSIGNED
logstash-2017.11.17 2 p STARTED 273766 162.4mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 2 r UNASSIGNED
logstash-2017.11.17 3 p STARTED 273367 163.5mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 3 r UNASSIGNED
logstash-2017.11.17 1 p STARTED 273632 164.1mb 127.0.0.1 TJhnG3M
logstash-2017.11.17 1 r UNASSIGNED
logstash-2017.11.17 4 p STARTED 273793 163mb 127.0.0.1 TJhnG3M

logstash-2017.11.08 2 p STARTED 245968 139.7mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 2 r UNASSIGNED
logstash-2017.11.08 3 p STARTED 246007 139.9mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 3 r UNASSIGNED
logstash-2017.11.08 1 p STARTED 246933 140.8mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 1 r UNASSIGNED
logstash-2017.11.08 4 p STARTED 246712 140.4mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 4 r UNASSIGNED
logstash-2017.11.08 0 p STARTED 246922 140mb 127.0.0.1 TJhnG3M
logstash-2017.11.08 0 r UNASSIGNED
logstash-2018.01.05 2 p STARTED 137960 91.5mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 2 r UNASSIGNED
logstash-2018.01.05 1 p STARTED 137454 91.1mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 1 r UNASSIGNED
logstash-2018.01.05 3 p STARTED 137453 90.9mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 3 r UNASSIGNED
logstash-2018.01.05 4 p STARTED 137905 91.2mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 4 r UNASSIGNED
logstash-2018.01.05 0 p STARTED 137419 90.9mb 127.0.0.1 TJhnG3M
logstash-2018.01.05 0 r UNASSIGNED
logstash-2017.10.24 2 p STARTED 91806 67mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 2 r UNASSIGNED
logstash-2017.10.24 3 p STARTED 91519 66.9mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 3 r UNASSIGNED
logstash-2017.10.24 1 p STARTED 91292 66.7mb 127.0.0.1 TJhnG3M
logstash-2017.10.24 1 r UNASSIGNED

Christian_Dahlqvist · February 28, 2018, 5:17am

It looks like you have a single node with a lot of small shards, which can be every inefficient and result in a lot of overhead. Read this blog post for some guidelines on how many shards you should aim to have in your cluster to get the most from it.

system · March 28, 2018, 5:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.