To find the problem you should take a look at the heap dump in a tool like MAT. If you find yourself spending more than a minute trying to work out what is using all the memory then give up. It should be front and centre on the main page. -- leak candidate #1.
See this thread for an example of what you should see.