Kibana dreaded timeout on dashboard

sorry but... do you have a little bit more explanation ?

I can get 15minutes 7 visualization. at once. no problem at all. up to 12 hours in fact.. ( most of the time ) BUT cannot go past it ...

What do you want me to do exactly or "provide" ?

If you request a few hours worth of data, you should be able to view the statistics (which includes time the aggregation took to ran) for each visualisation from within Kibana. If you look at the different visualisations and how long they took, is there any that stands out?

ill try to give it a shot.
Using F12 developper mode ? any specific tabs i should provide you information from ?

btw, thank you

In each visualisation there is an upwards arrow in the lower left corner. If you click on this you should see a button that says 'Statistics', which will provide this information.

12 hours

and a last one that is simply a "search view" in my dashboard ( so no stats )

It looks like you have some visualisations that require a lot of computation, especially the bottom one. If you are maxing out all CPU cores on both machines while querying, you may have hit the limit for what your cluster can handle, and may need to either try to make your dashboards less computationally intensive or scale out.

IF scaling out is not an option .. what option left do I have ?
btw. i just "removed" the botton visualisation, and tried to load the last 7days,

its a no go..

24hours, same thing.
12hours same thing .. so right now. cpu is loaded the hell up !

cpu usage 23% one nodes load av. 14, 10% the other one load av. 8

once the load dropped. I could load a view of the last 24hours.

How long is your retention period?

we plan, to be able to visualize up to three month of data. ( one month at once if needed.. but up to 3month active ) 1 year in archives

Upgrading to the latest 5.x release might actually help you. The 5.x releases have improved how caching works for indices covered entirely by the time period queried. If you used indices (with a single primary shard) that covered a smaller time period, e.g. a few hours, a good portion of the indices would be able to cache results for queries spanning longer time periods, resulting in faster response times. There is even a new rollover API that would allow you to create indices of a certain size irrespective of time period which may make this even easier to manage. The drawback with this approach is naturally that you may end up with a larger number of shards, which could become a problem for long term retention.

ok, so .. if I understand correctely . so long so far... im fuck*d ?

so .. how would I calculate adequately what I would need in term of CPU power ? if ever i was to scale horizontaly ... ?

Why is upgrading not an option?

im running Siemonster Stack. its sort of all in one bundle with ossec etc...
so I assume its version dependant + you said that in the long term, I would have a problem with number of shards...

If we assume 4 hours per index with 1 primary and 1 replica shard, you would get 360 shards per month. With that 3 months online retention should be fine with your current setup. You may even be able to stretch it quite a bit further, so I would not worry about that.

Apart from upgrading or making the visualisations more light-weight, I do not really have any good suggestions at the moment. Maybe someone else will chime in?

so my problem right now is how my "shards" are made ?

Shard size does affect query latency, so if your CPU is not saturated during querying, you may benefit from decreasing the shard size, e.g. by increasing the number of primary shards to 4 or 6 instead of 3. To benefit from indices covering a smaller time period I do however suspect you need to migrate to 5.x.

kk, it helps. thank you

still I just did a odd test ( for the fun ) . tried to fetch "only in discover" the last 30 days, and it work . no timeout but takes long to load..

so the problems is really upon visualisation ...

Discover just fetches a sample of 500 documents and do not aggregate across all records, so has little in common with the aggregations behind visualizations.

got it.

So long story made short... I would have to change the amount of shards to 6 and it will manage the "spread" by it self ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.