Kibana showing status error red

Elastic Stack Elasticsearch
1

Hi,
I am using Kibana version 4.6.3. ES version 2.4.3

I have a twitter index and another index called logdev for a different application. Each of the logdev indices have 63.5Mi data each day starting 12.15.2016 to 12.27.2016. I created a dashboard with the logdev indices and when i try to load the dashboard from date 12.15.2016 to 12.27.2016 i get "Request Timeout after 30000ms".

After a couple of minutes it changed into a Kibana Status error "RED".. (see below pic)

kiba status.PNG1314×494 24.2 KB

I know the problem here.. the heap size went beyond the total heap size available but I am not sure why it did that..how can i fix this problem?

1 Like
2

I would check the log files to see what the errors actually are. But I always thought this RED state occurred when the elastic search cluster is red. (At least that is always what happens to me)

Fist step it to know what the problem actually is then we can go further.

3

By logfiles, do you mean the stuff located under /var/log/elasticsearch? or logging,yml?

4

Well depends, not sure where you have the log files deploying to but if it is kibana there should be a log file for that.

If it is elasticsearch, that is in RED then probably just have to see which Node is down and look at it's log file.

5

I am still seeing the same error. I tried increasing the heap size from 1g to 6g..I tried adding another node to the cluster but still the visualization failing on long period searches...Please help!

I set bootsrap.memory_lock to true and increased heap to 6g..and now I am seeing this when i try to search 1 week data..it's not even a lot of data..why is dashbaord visualization failing?

6

ok, so your having a problem when actually searching . Why don't you describe to us the environment you have,

IE, how many servers (cpu, mem and disk space)
How many master,data and client nodes

Size of each index and how many shards.

If you can search for small amount of time, then cant search for a week then its probably the size and/or configuration of what you have.

You query is timing out, you could also up the timeout of kibana.ini to allow longer then 30 seconds. But most likely can fix it by fixing your environment.

7

Hi,
Thanks for your reply.

On the discover tab, the data loads up fine without any Timeout errors. However, when I try to load up all the data in the Dashboard tab, I get Timeout after 30000 ms.

Currently, I have ELK stack running in 1 remote server.

Server info:
Architecture: x86_64
Total Used Free
Mem: 3924392 2433660 1490732

Size of each index: 130MB
number of indices: 9
ELk Stack constist of 1 cluster 1 node.
Shards: 5

Things I tried:

  1. i tried adding an additional node...Didn't help
  2. i tried increasing the heap size 6g and locking the memory with boostrap.memory_lock:true -- didn't help
8

This is going to come down to the performance of your system not so much the Elasticsearch tuning, by default I find it will use every piece of resource a system has

So your search about 1GB of data on one node, yah that is going to be really slow,

So you have a 4GB sytem and a heap of 6GB yah your going to swap which will slow down your system badly

What is the speed of your Hard Drives (5k 7K 15K? SSD ) or is this in amazon?

So first reduce the size of your HEAP to say 2GB and spawn up 2 or three more ES nodes.

You want to have Spare unused memory for Caching with will increase your Search spreads as it will cache the Index data in to memory

The multiple nodes will give you more CPU's to search each shard, and if you have really slow disks 7k or less it will speed up your operations.

After that, if the speed is still slow, you can go back to my first suggesting and change the Timeout value in the kibana.yml from 30 seconds to 300 seconds or your preference.

Also to try to find out how slow your query's are try running a CURL request of your search instead of going though kibana, this will tell you truelly how slow/fast the query is as ELK will not timeout as fast

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.