Node Hot thread api gives following input
::: {appsec-es-master-0}{AAw21yu1Q1e5ZWMoBe2ZPA}{4IRnxBeST3ODTCHlLq_FqA}{10.244.3.48}{10.244.3.48:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}
Hot threads at 2019-10-19T08:13:45.483Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (59.9micros out of 500ms) cpu usage by thread 'ticker-schedule-trigger-engine'
10/10 snapshots sharing following 2 elements
java.base@12.0.1/java.lang.Thread.sleep(Native Method)
org.elasticsearch.xpack.watcher.trigger.schedule.engine.TickerScheduleTriggerEngine$Ticker.run(TickerScheduleTriggerEngine.java:193)
::: {appsec-es-master-2}{1QhuWcAaSni6YXA-sYOZhA}{s_fab2RrTgynFgGbf0paQA}{10.244.2.80}{10.244.2.80:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}
Hot threads at 2019-10-19T08:13:50.153Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (40.6micros out of 500ms) cpu usage by thread 'ticker-schedule-trigger-engine'
10/10 snapshots sharing following 2 elements
java.base@12.0.1/java.lang.Thread.sleep(Native Method)
org.elasticsearch.xpack.watcher.trigger.schedule.engine.TickerScheduleTriggerEngine$Ticker.run(TickerScheduleTriggerEngine.java:193)
::: {appsec-es-master-1}{gSxXRZG-SqC_i0GWGvAxgg}{y_Sgn76wQU27Ly3hqX0v7Q}{10.244.11.73}{10.244.11.73:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}
Hot threads at 2019-10-19T08:13:57.155Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
Following Error in the spark logs
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 477417.0 failed 4 times, most recent failure: Lost task 0.3 in stage 477417.0 (TID 1048412, 10.244.8.111, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [appsec_destination_owasp] failed; server[10.244.3.48:9200] returned [429|Too Many Requests:]
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:469)