I'm investigating a performance problem we've been noticing with one of our datanodes. Specifically es-data-01a.lab1. The /_cat/nodes API shows indexing.index_count hovering around 20, but the dashboard in Kibana shows a much lower number (0-5).
/_cat/nodes output
Same output in text:
Every 2.0s: ./esl GET '_cat/nodes?v&h=name,index*,segments.count,segments.index_writer_memory&s=name' unagi: Sat Jun 6 09:16:05 2020
name indexing.delete_current indexing.delete_time indexing.delete_total indexing.index_current indexing.index_time indexing.index_total indexing.index_failed segments.count segments.index_writer_memory
lab-rdu-es-data-01a 0 0s 0 23 60.7d 22030282842 135 8752 343.5mb
lab-rdu-es-data-01b 0 1.7s 49405 0 51.9d 18137685493 107 8428 292.1mb
lab-rdu-es-data-01c 0 189ms 323 5 62.6d 21783227217 1163 8584 243.9mb
lab-rdu-es-data-01d 0 62ms 210 5 81.9d 17506647734 179 8228 290.9mb
lab-rdu-es-data-01e 0 43ms 1 3 30.2d 5993549694 0 8587 266.3mb
lab-rdu-es-data-01f 0 3.1s 49390 3 26.5d 5036224438 0 8338 328.6mb
lab-rdu-es-data-01g 0 382ms 99 0 31.4d 6195974875 2 8550 659.3mb
lab-rdu-es-master-01a 0 0s 0 0 0s 0 0 0 0b
lab-rdu-es-master-01b 0 0s 0 0 0s 0 0 0 0b
lab-rdu-es-master-01c 0 0s 0 0 0s 0 0 0 0b
lab-rdu-es-ml-01a 0 0s 0 0 0s 0 0 0 0b
lab-rdu-es-ml-01b 0 0s 0 0 0s 0 0 0 0b
Kibana dashboard
/_cat/indicies
Every 2.0s: ./esl GET '_cat/indices?pretty&v&h=index,indexing.index*&s=indexing.index_current:desc' -H 'Accept: application/json' | jq .[] | paste - - - - - - - | gsed 's/indexing\.//g;s/[}{]\+[ \t]\+//... unagi: Sat Jun 6 10:05:41 2020
index: filebeat-flow-2020.06.06-14, index_current: 2, index_time: 4m, index_total: 1080291, index_failed: 0
index: metricbeat-6.5.1-2020.06.06, index_current: 2, index_time: 1.2d, index_total: 417630939, index_failed: 0
index: filebeat-ilm-6.5.1-2020.06.05-000009, index_current: 1, index_time: 3.1d, index_total: 661193295, index_failed: 0
index: .reporting-2020.02.02, index_current: 0, index_time: 0s, index_total: 0, index_failed: 0
index: .reporting-2020.01.26, index_current: 0, index_time: 0s, index_total: 0, index_failed: 0
index: heartbeat-7.5.1-2020.05.14, index_current: 0, index_time: 31.1m, index_total: 10359892, index_failed: 0
index: heartbeat-7.5.1-2020.05.13, index_current: 0, index_time: 31.2m, index_total: 10372263, index_failed: 0
index: heartbeat-7.5.1-2020.05.12, index_current: 0, index_time: 22.6m, index_total: 10390919, index_failed: 0
index: heartbeat-7.5.1-2020.05.11, index_current: 0, index_time: 0s, index_total: 0, index_failed: 0
index: heartbeat-7.5.1-2020.05.10, index_current: 0, index_time: 23m, index_total: 10386600, index_failed: 0
CMDs
$ watch -d "./esl GET '_cat/nodes?v&h=name,index*,segments.count,segments.index_writer_memory&s=name'"
$ watch -d "./esl GET '_cat/indices?pretty&v&h=index,indexing.index*&s=indexing.index_current:desc' -H 'Accept: application/json' | jq .[] | paste - - - - - - - | gsed 's/indexing\.//g;s/[}{]\+[ \t]\+//g;s/[ \t]\+}//g;s/\"//g' | column -t | head"