Hello, we have our cluster currently set up across 2 machines, with 1 machine having 1 data+master node, and the other machine having a data only node as well as 2 master nodes. The data only node, as well as the data+master node, are configured to have a reserved JAVA heap size of 32 GB. The 2 master nodes each have 16GB reserved heap size. Pretty frequently, I notice that the data only node will fail every so often, maybe every hour or so? When it fails, I see the error:
[2015-08-12 14:33:50,813][WARN ][netty.channel.DefaultChannelFuture] An exception was thrown by ChannelFutureListener.
java.lang.OutOfMemoryError: unable to create new native thread
I'm not sure why this is, as I can see that there are about 6GB used for fielddata, and that the used heap size % is at about 50%...I see a lot of other error messages, but they're all basically giving me the same outofmemory error message. I can't figure out why this node is crashing, especially since it's essentially the same as the data+master node in terms of configuration, and if anything that node should be using more memory than the data only node! Can anybody help?