I have found something similar. Although both my -Xmx and Initial/Maximum memory pool values are set to 4GB, when indexing as few as 100K documents, I find the memory just climbs up and up, consuming all available (and then some) RAM. It's a bit of a big deal, what the heck is the point of having a min/max value if it totally gets ignored and can bring a machine to it's knees anyway? I've seen the ES process consuming >16GB of RAM, despite the 4GB setting.
What can I do to stop this? It's a bit problematic to put this on a production machine when it can kill the machine, despite specific settings designed to protect the OS and any other services.
CraigL