Good morning.
I'm running some tests on using docker to spin up multiple nodes and cluster them together on a single bare metal server.... but having major query performance issues.
For my test I only have 2 nodes with 24GB each. I have specified node.processors=8 as the box only has 16 cores. (I only added this setting later - will explain)
The heap on either node hardly goes above 60%
CPU load on the box is a bit high ..but not ridiculously so.
Disks does not seem to indicate any io problems.
I've started Stack Monitoring on both nodes and the the search latency sits pretty constant at 7-10ms on both nodes.
The only thing I found was that on the second (and always only on the second node) ...the search threads (from _cat/thread_pool) sits with a huge queue.
After I bounce the DB everything is super fast and responsive m but then degrades over a few hours to the point where a query that usually takes milliseconds to return , takes like 10s - and at this point I can see the threads queued on the second node.
(And even in that state , the stack monitoring doesnt show any degredation in search latency)
Because of this I tried bumping the search threads ... which didnt help and then I added the processors setting thinking that first node might be taking priority (in terms of processing) over the second one. But that also didnt help .
Any ideas ?
Is it viable to run a cluster like that ?
Is it still a thing to not configure your heap above 32GB ? (Which was sort of the idea behind running them as dockers ... instead of having to spin up a bunch of VMs)
More Info:
Running ES 7.10
2 nodes with 24GB heap each. (The box has 72GB)
"Default" config for both nodes ...with both being master eligible - but ES01 Is the master and serves all requests.
All indexes configured with 2 shards and 0 replicas. (So most of the indexes are split evenly across both the nodes)
Thanx in advance.