I'm using ElasticCloud to host ES.
I'm seeing 'some' slow requests coming from my Azure Web App talking to my ES cluster.
- Azure web app: Azure East (Sydney)
- ES cluster: AWS Sydney, 1 node, 1 primary shard, 0 replicas (yes.. i know not ideal, but this is just a dev/test cluster)
So we're talking 15ms of latency between Azure and AWS.
99% of requests are < 50ms.. but all of a sudden i get a bit of a spike that lasts a couple of minutes.
Here's an example from ApplicationInsights in Azure:
So that's saying it took 914ms to do the network call.
However, in Kibana.. i can't see this request at all.
I'm looking at the 'Search Latency' under 'Overview'.. and would expect to find a spike there.. but it never goes above 0.36ms.
Given the network latency is < 50ms.. something is happening.. either a network blip, or something on the ES side.
Maybe it's not the query that's slow.. but something before/after that? (for example: maybe connecting to all the nodes, getting results etc)?.
Any tips on tracking down these slow requests? Is there another place i can look in Kibana? (note: i can't use slowlog.. as that's not supported by ElasticCloud)