How to track down slow queries?

[rpm1984]

Hi there,

I'm using ElasticCloud to host ES.

I'm seeing 'some' slow requests coming from my Azure Web App talking to my ES cluster.

Setup:

• Azure web app: Azure East (Sydney)
• ES cluster: AWS Sydney, 1 node, 1 primary shard, 0 replicas (yes.. i know not ideal, but this is just a dev/test cluster)

So we're talking 15ms of latency between Azure and AWS.

99% of requests are < 50ms.. but all of a sudden i get a bit of a spike that lasts a couple of minutes.

Here's an example from ApplicationInsights in Azure:

So that's saying it took 914ms to do the network call.

However, in Kibana.. i can't see this request at all.

I'm looking at the 'Search Latency' under 'Overview'.. and would expect to find a spike there.. but it never goes above 0.36ms.

Given the network latency is < 50ms.. something is happening.. either a network blip, or something on the ES side.

Maybe it's not the query that's slow.. but something before/after that? (for example: maybe connecting to all the nodes, getting results etc)?.

Any tips on tracking down these slow requests? Is there another place i can look in Kibana? (note: i can't use slowlog.. as that's not supported by ElasticCloud)

Thanks!

[forloop (Russ Cam)]

Are you making the requests from Azure web app with NEST?

If so, would be worth enabling network tracing, to see where time is being spent: https://docs.microsoft.com/en-us/dotnet/framework/network-programming/network-tracing

This should some insight into whether there's something related to the network, or not.