Thread waitedTime is more while connecting to elastic

[Sumaiya (Saumaiya)]

Hello All ,

One of our spring boot micro service connecting to elastic has high response time , while the took time itself is 50ms the service response time is high . After taking couple of threaddumps I see there are in total 10 threads in each dump and 1 or 2 out of 10 threads are in RUNNABLE status always and remaining threads are either in WATING or TIMED_WATING. The below thread says waitedTime after thread contention monitoring was enable and it shows waitedTime": 4985607ms .

1. What should be my starting point for analysis?
2. What could cause these threads to wait for so long ?

Appreciate your help on this .

2 threads has this status

`

{
"threadName": "http-nio-8080-exec-12",
"threadId": 37,
"blockedTime": 0,
"blockedCount": 0,
"waitedTime": 4985607,
"waitedCount": 1245,
"lockName": null,
"lockOwnerId": -1,
"lockOwnerName": null,
"inNative": false,
"suspended": false,
"threadState": "RUNNABLE",
"stackTrace": [
{
"methodName": "dumpThreads0",
"fileName": "ThreadImpl.java",
"lineNumber": -2,
"className": "sun.management.ThreadImpl",
"nativeMethod": true
},
{
"methodName": "dumpAllThreads",
"fileName": "ThreadImpl.java",
"lineNumber": 454,
"className": "sun.management.ThreadImpl",
"nativeMethod": false
},
{
"methodName": "invoke",
"fileName": "DumpEndpoint.java",
"lineNumber": 44,
"className": "org.springframework.boot.actuate.endpoint.DumpEndpoint",
"nativeMethod": false
},
{
"methodName": "invoke",
"fileName": "DumpEndpoint.java",
"lineNumber": 31,
"className": "org.springframework.boot.actuate.endpoint.DumpEndpoint",
"nativeMethod": false
},

`
8 threads have this status

{
"threadName": "http-nio-8080-exec-14",
"threadId": 67,
"blockedTime": 0,
"blockedCount": 0,
"waitedTime": 4756546,
"waitedCount": 1248,
"lockName": null,
"lockOwnerId": -1,
"lockOwnerName": null,
"inNative": false,
"suspended": false,
"threadState": "RUNNABLE",
"stackTrace": [
{
"methodName": "dumpThreads0",
"fileName": "ThreadImpl.java",
"lineNumber": -2,
"className": "sun.management.ThreadImpl",
"nativeMethod": true
},
{
"methodName": "dumpAllThreads",
"fileName": "ThreadImpl.java",
"lineNumber": 454,
"className": "sun.management.ThreadImpl",
"nativeMethod": false
},
{
"methodName": "invoke",
"fileName": "DumpEndpoint.java",
"lineNumber": 44,
"className": "org.springframework.boot.actuate.endpoint.DumpEndpoint",
"nativeMethod": false
},
{
"methodName": "invoke",
"fileName": "DumpEndpoint.java",
"lineNumber": 31,
"className": "org.springframework.boot.actuate.endpoint.DumpEndpoint",
"nativeMethod": false

[Sumaiya (Saumaiya)]

I did profiling of my client code which is calling elastic . Most of time is spent on Class CountDownLatch