Kibana - No Instances Running

Elastic Orchestration Elastic Cloud Enterprise (ECE)
#1

Hi,

All of a sudden my kibana deployment disappeared. There's not much info in the UI - has anyone seen this before? Any tips on resolution?

I have terminated the deployment, re-created it, tried scaling up to more nodes, and more, however each time ends up in a failed kibana deployment.

Screenshot%20from%202019-10-18%2010-45-14
Screenshot from 2019-10-18 10-45-14.png1898×946 75.3 KB

Thanks, Justin

#2

Hi, sorry you're having issues. It looks like you are running on Elasticsearch Service, which has a separate help ticketing system - https://cloud.elastic.co/help where you can post your cluster is and someone can look in more detail than is possible here (also this forum is for the the on-prem - "enterprise" version of the product)

#3

I had a quick look in the logs (annoying the Kibana logs are both not very useful without changing the config and not accessible to ESS customers - both things we're actively working on) and the only thing I could see was all shards failed: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [1024]. This limit can be set by changing the [search.max_open_scroll_context] setting. - is it possible you were using some really heavy Kibana activity (can't imagine what it could have been) and it brought the service down? I see it's now happy after increase in number of zones and capacity

Anyway if you see the problem again, create a support ticket as above, and an admin can add the extra logging params and see if that gives more insight into the underlying issue

#4

Thanks @Alex_Piggott for digging into this. I agree seeing more logs would be helpful!

We have a query CLI to allow people to tail logs which uses scrolls, but this hits the ES api directly. Would this exception thrown by ES cause kibana to crash?

Best, Justin

#5

Interesting, the error I saw was definitely in the Kibana logs (I actually didn't check the ES logs since I assumed you'd had a look in there for anything relevant)

Alex

#6

Must be a coincidence then. Would a large number of people viewing a dashboard cause this issue? Or perhaps a dashboard with a high refresh rate? Or a combination?

Best, Justin

#7

I've never seen it before, actually looking into it a bit more, that error is from ES: eg Elasticsearch 7.0.1 - Trying to create too many scroll contexts. Must be less than or equal to: [500] .. so presumably your client was using up all the scroll contexts and via some unknown mechanism this caused errors in Kibana.

There isn't sufficient information logged currently to tell if that's related to the loss of availability of Kibana unfortunately. But at least this suggests that you might want to have a look into your client behavior?

#8

Will do. Swapping the cli to use to/from pagination vs. creating a scroll context. Hopefully that helps out. Best, Justin

#9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.