Which version in the 6.4 series were you running before the upgrade?
Can you take a heap dump?
It's possible it's the same issue as reported here: Post-upgrade issues: 6.4.2 ES heap running away (6.5.0 too) but a heap dump is the only way to be sure. The issue was introduced in 6.4.1 and the fix will be out in 6.5.3 if so.