Why my bulk request not using all cpus of others node?

Hi Team,

Below is my output of thread pool. I dont know why my only one node'cpu is being use for bulk request.

curl -X GET "myhost:9200/_cat/thread_pool/?v"
node_name     name                active queue rejected
node1       bulk                    24  4522        0
node1       fetch_shard_started      0     0        0
node1       fetch_shard_store        0     0        0
node1       flush                    1     0        0
node1       force_merge              0     0        0
node1       generic                  0     0        0
node1       get                      0     0        0
node1       index                    0     0        0
node1       listener                 0     0        0
node1       management               1     0        0
node1       refresh                  2     0        0
node1       search                  21     0        0
node1       snapshot                 0     0        0
node1       warmer                   0     0        0
node2       bulk                     0     0        0
node2       fetch_shard_started      0     0        0
node2       fetch_shard_store        0     0        0
node2       flush                    0     0        0
node2       force_merge              0     0        0
node2       generic                  0     0        0
node2       get                      0     0        0
node2       index                    0     0        0
node2       listener                 0     0        0
node2       management               1     0        0
node2       refresh                  0     0        0
node2       search                   0     0        0
node2       snapshot                 0     0        0
node2       warmer                   0     0        0
node3       bulk                     3     0        0
node3       fetch_shard_started      0     0        0
node3       fetch_shard_store        0     0        0
node3       flush                    0     0        0
node3       force_merge              0     0        0
node3       generic                  0     0        0
node3       get                      0     0        0
node3       index                    0     0        0
node3       listener                 0     0        0
node3       management               1     0        0
node3       refresh                  0     0        0
node3       search                   0     0        0
node3       snapshot                 0     0        0
node3       warmer                   0     0        0

Are you sending all bulk requests to just one node? Are the nodes configured the same?

In connection i have mention all nodes. So it is making connection to all nodes. All three nodes are master nodes with no replication. By default it is round robin selector. I think it should distribute the traffic to all nodes. But it is not happening which causing my write slow.

I am using php-sdk for bulk write.

Can you share the output of GET _cat/shards?

Here is my _cat/shards

index3-201903           4 p STARTED  11453505    3.9gb ip  node1
index3-201903           1 p STARTED  11453637      4gb ip  node2
index3-201903           2 p STARTED  11448352      4gb ip  node3
index3-201903           3 p STARTED  11444246      4gb ip  node2
index3-201903           0 p STARTED  11445181      4gb ip  node3
index7-201903           4 p STARTED   9324240    3.2gb ip  node1
index7-201903           1 p STARTED   9320367    3.1gb ip  node2
index7-201903           2 p STARTED   9323628    3.2gb ip  node1
index7-201903           3 p STARTED   9324646    3.2gb ip  node2
index7-201903           0 p STARTED   9319057    3.2gb ip  node3
index1-201906           1 p STARTED         0     230b ip  node3
index1-201906           4 p STARTED         0     230b ip  node2
index1-201906           2 p STARTED         0     230b ip  node2
index1-201906           3 p STARTED         0     230b ip  node1
index1-201906           0 p STARTED         0     230b ip  node3
index9-201904           1 p STARTED   5946930    2.4gb ip  node3
index9-201904           4 p STARTED   5946084    2.4gb ip  node1
index9-201904           2 p STARTED   5946722    2.5gb ip  node1
index9-201904           3 p STARTED   5945667    2.4gb ip  node2
index9-201904           0 p STARTED   5946815    2.5gb ip  node3
index2-201902           1 p STARTED  12096140    4.8gb ip  node1
index2-201902           4 p STARTED  12091361    4.6gb ip  node2
index2-201902           2 p STARTED  12091717    4.8gb ip  node2
index2-201902           3 p STARTED  12100938    4.8gb ip  node1
index2-201902           0 p STARTED  12091226    4.7gb ip  node3
index9-201906           1 p STARTED         0     230b ip  node1
index9-201906           4 p STARTED         0     230b ip  node2
index9-201906           2 p STARTED         0     230b ip  node3
index9-201906           3 p STARTED         0     230b ip  node1
index9-201906           0 p STARTED         0     230b ip  node3
index4-201903           1 p STARTED  17308612    5.4gb ip  node3
index4-201903           4 p STARTED  17298300    5.5gb ip  node2
index4-201903           2 p STARTED  17305067    5.2gb ip  node2
index4-201903           3 p STARTED  17296242    5.5gb ip  node1
index4-201903           0 p STARTED  17297445    5.2gb ip  node3
index6-201905           4 p STARTED   8241309    2.1gb ip  node1
index6-201905           1 p STARTED   8236951    2.3gb ip  node2
index6-201905           2 p STARTED   8236060    2.6gb ip  node3
index6-201905           3 p STARTED   8237484    2.1gb ip  node2
index6-201905           0 p STARTED   8238824    2.4gb ip  node3
index7-201905           1 p STARTED   3291973    1.6gb ip  node3
index7-201905           4 p STARTED   3287984    1.2gb ip  node2
index7-201905           2 p STARTED   3291085      1gb ip  node2
index7-201905           3 p STARTED   3293073      1gb ip  node1
index7-201905           0 p STARTED   3290959    1.2gb ip  node3
index4-201906           4 p STARTED         0     230b ip  node1
index4-201906           1 p STARTED         0     230b ip  node2
index4-201906           2 p STARTED         0     230b ip  node3
index4-201906           3 p STARTED         0     230b ip  node2
index4-201906           0 p STARTED         0     230b ip  node3
index9-201902           4 p STARTED   8548210    2.9gb ip  node1
index9-201902           1 p STARTED   8559473    2.9gb ip  node2
index9-201902           2 p STARTED   8554209    2.8gb ip  node3
index9-201902           3 p STARTED   8550218    2.8gb ip  node2
index9-201902           0 p STARTED   8555954    2.9gb ip  node3
index9-201905           1 p STARTED   2189115    1.3gb ip  node3
index9-201905           4 p STARTED   2186322  849.1mb ip  node2
index9-201905           2 p STARTED   2187710      1gb ip  node2
index9-201905           3 p STARTED   2187424  904.5mb ip  node1
index9-201905           0 p STARTED   2187248  969.9mb ip  node3
index2-201904           1 p STARTED  24682711   10.7gb ip  node1
index2-201904           4 p STARTED  24691660   10.7gb ip  node2
index2-201904           2 p STARTED  24688984   10.9gb ip  node3
index2-201904           3 p STARTED  24670928   10.8gb ip  node1
index2-201904           0 p STARTED  24690389     11gb ip  node3

Hmm ok that looks like a sensible spread of shards across the nodes.

Can you try sending traffic just to node2 and see if this moves the load away from node1 or not?

Hi David,

Still All the traffic is routing to one node only.

Does the PHP client connect to the nodes in the order they are specified, leading to all threads using the first node in the list?

Sure, but which node? If you're sending traffic to node2 is it still getting stuck on node1?

yes exactly. All traffic is routing to node 1 only. Some active thread i have seen while continuously hitting _cat/threa_pool But most of the time node1 only receiving all traffic. My writing is continuously going on.

node_name                name                active queue rejected
node1                  bulk                    24  6753        0
node1                  fetch_shard_started      0     0        0
node1                  fetch_shard_store        0     0        0
node1                  flush                    0     0        0
node1                  force_merge              0     0        0
node1                  generic                  0     0        0
node1                  get                      0     0        0
node1                  index                    0     0        0
node1                  listener                 0     0        0
node1                  management               2     0        0
node1                  refresh                  2     0        0
node1                  search                   0     0        0
node1                  snapshot                 0     0        0
node1                  warmer                   0     0        0
node2                  bulk                     3     0        0
node2                  fetch_shard_started      0     0        0
node2                  fetch_shard_store        0     0        0
node2                  flush                    0     0        0
node2                  force_merge              0     0        0
node2                  generic                  0     0        0
node2                  get                      0     0        0
node2                  index                    0     0        0
node2                  listener                 0     0        0
node2                  management               1     0        0
node2                  refresh                  0     0        0
node2                  search                   0     0        0
node2                  snapshot                 0     0        0
node2                  warmer                   0     0        0
node3                  bulk                    10     0        0
node3                  fetch_shard_started      0     0        0
node3                  fetch_shard_store        0     0        0
node3                  flush                    0     0        0
node3                  force_merge              0     0        0
node3                  generic                  0     0        0
node3                  get                      0     0        0
node3                  index                    0     0        0
node3                  listener                 0     0        0
node3                  management               2     0        0
node3                  refresh                  0     0        0
node3                  search                   0     0        0
node3                  snapshot                 0     0        0
node3                  warmer                   0     0        0

I suspect there's something wrong with node1 causing it to process traffic much slower than the other two nodes.

Can you run the following command:

GET /_nodes/stats?filter_path=nodes.*.name,nodes.*.indices.indexing.index_total

Then do some indexing for a while and finally run the same command again:

GET /_nodes/stats?filter_path=nodes.*.name,nodes.*.indices.indexing.index_total

This will tell us whether that node is really seeing more traffic than the other two.

Hi david,
Here is response:

curl -XGET "myhost:9200/_nodes/stats?filter_path=nodes.*.name,nodes.*.indices.indexing.index_total&pretty"
{
  "nodes" : {
	"oUIhmmUZRL-adjPXWQjx5Q" : {
	  "name" : "node3",
	  "indices" : {
	    "indexing" : {
	      "index_total" : 4345928590
	    }
	  }
	},
	"9VMMk-kRRjWKDKB92XGTbA" : {
	  "name" : "node1",
	  "indices" : {
	    "indexing" : {
	      "index_total" : 123063502
	    }
	  }
	},
	"ybo_Txv9RICBosQ4QflRqw" : {
	  "name" : "node2",
	  "indices" : {
	    "indexing" : {
	      "index_total" : 4006695200
	    }
	  }
	}
  }
}

The number of node2 and node3 more then node 1. I guess this is total indexing number since uptime. And i recently added node1 (2 days ago). This is might be reason of less index_total

That's why I asked you to run that command twice.

M sorry david, My bad. Here is another response:

	curl -XGET "myhost:9200/_nodes/stats?filter_path=nodes.*.name,nodes.*.indices.indexing.index_total&pretty"
	{
		"nodes" : {
			"oUIhmmUZRL-adjPXWQjx5Q" : {
			  "name" : "node3",
			  "indices" : {
			    "indexing" : {
			      "index_total" : 4348430999
			    }
			  }
			},
			"9VMMk-kRRjWKDKB92XGTbA" : {
			  "name" : "node1",
			  "indices" : {
			    "indexing" : {
			      "index_total" : 125461151
			    }
			  }
			},
			"ybo_Txv9RICBosQ4QflRqw" : {
			  "name" : "node2",
			  "indices" : {
			    "indexing" : {
			      "index_total" : 4009182561
			    }
			  }
			}
		}
	}

Thanks, now we look at the differences:

node1:  125461151- 123063502 = 2397649
node2: 4009182561-4006695200 = 2487361
node3: 4348430999-4345928590 = 2502409

So it looks like node1 is actually handling slightly less traffic than the other two nodes. This does suggest that there's something different about that node.

Thanks David,

That makes sense but whenever i look to thread_pool it showing only node1 is active and shows some requests in active .

As per our analysis with indexing. It seems indexing rate is equal on all three nodes. But i am not sure about thread pools because in my monitoring graph also it shows highly bulk thread is using by only one node1.

Also one important point:

All three nodes's load average is different:

node1: 46.33, 43.81, 42.94
node2: 8.38, 8.40, 8.41
node3: 2.88, 3.35, 3.63

I also make sure there is no other service is running on node1.

Also whenever i increase my bulk write threads It increase my queue number of thread_pool of node1 only.

That is consistent with node1 processing indexing requests much slower than the other two nodes.

That could be because node1 has a much slower disk than the other two nodes.

Thanks David for explaination .

Hmm It might be. But i am running all three nodes on SATA disk. I know SSD is recommended. So what kind of disk slow can be possible according to you ?

I don't know, sorry, I'm not really in a position to diagnose performance issues in your IO subsystem.

Okay david no problem. Thanks for you valuable answer :slight_smile: