From some guidance, it suggests that master nodes do not need much cpu/memory. From my clusters, this looks true for most of the time. But sometimes, I do observe one of the master node has a high load_5m. e.g. 5 or 6, but it is just as normal as around 1-2 for the other two master nodes. But the cluster looks healthy and latency is also fine. Could someone suggest what kind of tasks master are doing to take such a high load? Would this potentially cause some issues for the cluster?
You should not be sending any queries or indexing requests to dedicated master nodes, which would be the first thing to check. Make sure clients do not access these nodes and that you do not get them added by enabling sniffing.
yep. master nodes are dedicated only to form the cluster for nodes discovery. The only query/indexing facing nodes are coordinating nodes. Even data nodes are behind coordinating nodes.
If it is just the current master node that is busy it could be large, complex or frequent updates to the cluster state may be the cause. Run the hot threads API to find out. This could be caused by shard creation or relocation and/or mappings that are frequently dynamically updated.
::: {elasticsearch-master-1}{Dlqb04jRScWu2QNNZB2isA}{R8t9nmnPToKG0vPkITNFRg}{10.2.59.9}{10.2.59.9:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-data-0}{qHO2LMWBTXaFDL3pL2DAXw}{LPoCTqzARF-0AA-SmjNGAQ}{10.2.99.10}{10.2.99.10:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-data-1}{yVxYoGZHQ9GnO5Wzf4vjuQ}{9InCENoISXCcp8zh65NrYg}{10.2.130.8}{10.2.130.8:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-master-0}{8eVYyj2iSe2tH4MAEAMgiw}{akkzhRZeR8KsY4wmH4WbRw}{10.2.99.9}{10.2.99.9:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.684Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-coordinating-2}{mGQr3wEYR-mF-fqlq7iKpw}{xJI1BezjR_aQTcdQ6bDTLQ}{10.2.100.8}{10.2.100.8:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.688Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-coordinating-0}{R4T3Qtd7Tc2QJNNcEEOkNg}{PsWeirOdS1qkRsZ_BuNoRg}{10.2.99.8}{10.2.99.8:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-data-2}{mljGgKZRTBClillQfmpz3w}{nrb1TepKQ4qVOZoQIndgRw}{10.2.83.7}{10.2.83.7:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-coordinating-1}{0O-dsg0HQMqdTw2skiB-bw}{p_skTi0gS36jc3W-iTkL0A}{10.2.8.7}{10.2.8.7:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.682Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:::: {elasticsearch-master-2}{y90TWMXQTmCTFSrowtrl9g}{XT36RsS8SnaZMW9og7SiUw}{10.2.8.10}{10.2.8.10:9300}{box_type=hot}
Hot threads at 2019-09-26T05:22:35.681Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
This cluster should not be very busy as it does not have much data to be ingested into.
It uses a daily index. For each index, it has 26Gb doc, 50M docs (storage size: 52GB as replica=1)
so far, it only has 5 indicies
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.