Kibana thrown a tantrum with Dashboard. ES timeout 30000ms

Elastic Stack Elasticsearch
1

Hello !

despite the title it is related to ES.
somehow my ELK stack stop working after 50 shards or so (46) (9 day 5shards per indices plus kibana. )
I have to close indices to get it working again, OR I get 30000ms timeout error ...

Heres the spec of the server so far.

4 vCPU
8 gig of ram
Centos 7 Latest

Kibana 4.5.3
Logstash 2.3.4
ES 2.3.4

Load average: 0.75, 1.05, 1.35
Ram usage average : 4.7gig out of 8

Need help, I plan to use that server for 30 to 60 indices. maybe more. and rotate monthly bi-monthly

2

Hi,

Out of curiosity what is the size of these shards (GET /_cat/shards).

Using one node for all the 50 shards means that the shards compete for the server resource (CPU but also memory), forcing the OS to load/evict page cache for example.

3

thank you for reply

curl -XGET 'localhost:9200/_cat/shards'
ftg-2016.07.31 3 p STARTED    408044 404.5mb 10.24.232.232 Whiplash
ftg-2016.07.31 3 r UNASSIGNED
ftg-2016.07.31 1 p STARTED    407935 403.8mb 10.24.232.232 Whiplash
ftg-2016.07.31 1 r UNASSIGNED
ftg-2016.07.31 2 p STARTED    408192   405mb 10.24.232.232 Whiplash
ftg-2016.07.31 2 r UNASSIGNED
ftg-2016.07.31 4 p STARTED    408206 405.2mb 10.24.232.232 Whiplash
ftg-2016.07.31 4 r UNASSIGNED
ftg-2016.07.31 0 p STARTED    408357 404.8mb 10.24.232.232 Whiplash
ftg-2016.07.31 0 r UNASSIGNED
ftg-2016.07.30 3 p STARTED    427295 427.1mb 10.24.232.232 Whiplash
ftg-2016.07.30 3 r UNASSIGNED
ftg-2016.07.30 1 p STARTED    425903 422.4mb 10.24.232.232 Whiplash
ftg-2016.07.30 1 r UNASSIGNED
ftg-2016.07.30 2 p STARTED    426876 426.4mb 10.24.232.232 Whiplash
ftg-2016.07.30 2 r UNASSIGNED
ftg-2016.07.30 4 p STARTED    427619   426mb 10.24.232.232 Whiplash
ftg-2016.07.30 4 r UNASSIGNED
ftg-2016.07.30 0 p STARTED    426298 425.9mb 10.24.232.232 Whiplash
ftg-2016.07.30 0 r UNASSIGNED
.kibana        0 p STARTED        22  97.7kb 10.24.232.232 Whiplash
.kibana        0 r UNASSIGNED
ftg-2016.07.28 3 p STARTED    856226   903mb 10.24.232.232 Whiplash
ftg-2016.07.28 3 r UNASSIGNED
ftg-2016.07.28 1 p STARTED    856394 900.1mb 10.24.232.232 Whiplash
ftg-2016.07.28 1 r UNASSIGNED
ftg-2016.07.28 2 p STARTED    855864   900mb 10.24.232.232 Whiplash
ftg-2016.07.28 2 r UNASSIGNED
ftg-2016.07.28 4 p STARTED    855602 899.5mb 10.24.232.232 Whiplash
ftg-2016.07.28 4 r UNASSIGNED
ftg-2016.07.28 0 p STARTED    855662 898.3mb 10.24.232.232 Whiplash
ftg-2016.07.28 0 r UNASSIGNED
ftg-2016.07.29 3 p STARTED    786799 836.8mb 10.24.232.232 Whiplash
ftg-2016.07.29 3 r UNASSIGNED
ftg-2016.07.29 1 p STARTED    786266 839.7mb 10.24.232.232 Whiplash
ftg-2016.07.29 1 r UNASSIGNED
ftg-2016.07.29 2 p STARTED    787000 842.4mb 10.24.232.232 Whiplash
ftg-2016.07.29 2 r UNASSIGNED
ftg-2016.07.29 4 p STARTED    787079 841.3mb 10.24.232.232 Whiplash
ftg-2016.07.29 4 r UNASSIGNED
ftg-2016.07.29 0 p STARTED    786925 840.4mb 10.24.232.232 Whiplash
ftg-2016.07.29 0 r UNASSIGNED
ftg-2016.07.27 3 p STARTED    534870 585.2mb 10.24.232.232 Whiplash
ftg-2016.07.27 3 r UNASSIGNED
ftg-2016.07.27 1 p STARTED    532778 581.4mb 10.24.232.232 Whiplash
ftg-2016.07.27 1 r UNASSIGNED
ftg-2016.07.27 2 p STARTED    533281 583.8mb 10.24.232.232 Whiplash
ftg-2016.07.27 2 r UNASSIGNED
ftg-2016.07.27 4 p STARTED    533747 584.5mb 10.24.232.232 Whiplash
ftg-2016.07.27 4 r UNASSIGNED
ftg-2016.07.27 0 p STARTED    533576 579.5mb 10.24.232.232 Whiplash
ftg-2016.07.27 0 r UNASSIGNED
ftg-2016.08.03 3 p STARTED    879636 945.4mb 10.24.232.232 Whiplash
ftg-2016.08.03 3 r UNASSIGNED
ftg-2016.08.03 1 p STARTED    879956 944.4mb 10.24.232.232 Whiplash
ftg-2016.08.03 1 r UNASSIGNED
ftg-2016.08.03 2 p STARTED    880111   942mb 10.24.232.232 Whiplash
ftg-2016.08.03 2 r UNASSIGNED
ftg-2016.08.03 4 p STARTED    879845 944.5mb 10.24.232.232 Whiplash
ftg-2016.08.03 4 r UNASSIGNED
ftg-2016.08.03 0 p STARTED    879493 950.2mb 10.24.232.232 Whiplash
ftg-2016.08.03 0 r UNASSIGNED
ftg-2016.08.04 3 p STARTED    381896   404mb 10.24.232.232 Whiplash
ftg-2016.08.04 3 r UNASSIGNED
ftg-2016.08.04 1 p STARTED    382116 408.4mb 10.24.232.232 Whiplash
ftg-2016.08.04 1 r UNASSIGNED
ftg-2016.08.04 2 p STARTED    382471   395mb 10.24.232.232 Whiplash
ftg-2016.08.04 2 r UNASSIGNED
ftg-2016.08.04 4 p STARTED    381581 397.7mb 10.24.232.232 Whiplash
ftg-2016.08.04 4 r UNASSIGNED
ftg-2016.08.04 0 p STARTED    383235 393.1mb 10.24.232.232 Whiplash
ftg-2016.08.04 0 r UNASSIGNED
ftg-2016.08.01 3 p STARTED    778014   844mb 10.24.232.232 Whiplash
ftg-2016.08.01 3 r UNASSIGNED
ftg-2016.08.01 1 p STARTED    779553 844.5mb 10.24.232.232 Whiplash
ftg-2016.08.01 1 r UNASSIGNED
ftg-2016.08.01 2 p STARTED    780779 847.2mb 10.24.232.232 Whiplash
ftg-2016.08.01 2 r UNASSIGNED
ftg-2016.08.01 4 p STARTED    779068 835.1mb 10.24.232.232 Whiplash
ftg-2016.08.01 4 r UNASSIGNED
ftg-2016.08.01 0 p STARTED    779956 841.2mb 10.24.232.232 Whiplash
ftg-2016.08.01 0 r UNASSIGNED
ftg-2016.08.02 3 p STARTED    862595 939.6mb 10.24.232.232 Whiplash
ftg-2016.08.02 3 r UNASSIGNED
ftg-2016.08.02 1 p STARTED    863466 939.4mb 10.24.232.232 Whiplash
ftg-2016.08.02 1 r UNASSIGNED
ftg-2016.08.02 2 p STARTED    862714 942.2mb 10.24.232.232 Whiplash
ftg-2016.08.02 2 r UNASSIGNED
ftg-2016.08.02 4 p STARTED    864493 945.3mb 10.24.232.232 Whiplash
ftg-2016.08.02 4 r UNASSIGNED
ftg-2016.08.02 0 p STARTED    864566 946.8mb 10.24.232.232 Whiplash
ftg-2016.08.02 0 r UNASSIGNED
4

see anything relevent in that ?

thank you

Ive removed all replicas shard.

curl -XPUT 'localhost:9200/.kibana/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}

curl -XPUT 'localhost:9200/ftg-*/_settings' -d '
{
"index" : {
"number_of_replicas" : 0
}
}

added a new device to output log. and now gets 30000ms error if trying to visualise data from last 7days.

5

Your indices are very small, you should use a single shard.
Is there anything in the logs of ES?

6

negative

nothing relevent no error at all.
can I change the number of shards im using "hot" ? and what would be the gain ?