We are running a 5 node cluster with 3 data nodes (where 1 is master eligible) and 2 master (non-data eligible nodes) as windows services.
Each of the data node has 6 GB allocated heap memory and runs on machines with 20 GB RAM available. (from startup log: heap size [5.8gb], compressed ordinary object pointers [true])
After a long period of stability we decided a couple of days ago to enable HTTP compression from the client we use for requesting ES data (NEST .net client). Later the same night we experienced unexpected shutdowns of the data nodes. This has happened several times since then. In the logs I can find that the shutdown is caused by a OOM (OutOfMemoryError: Direct buffer memory).
The monitoring graphs does not indicate that we are using all the allocated heap memory when the exception occurs:
Any help is highly appreciated!