Post-upgrade issues: 6.4.2 ES heap running away (6.5.0 too)

[DavidTurner]

@DavidTurner

I ask about this because if I understand your screenshot correctly the heap dump has 358,000+ instances of things like IndexSettings and IndexMetaData , which are created (once) when the first shard of an index is allocated to a node. They should be cleaned up when the last shard of that index is deallocated, but even if they weren't I can't see how Elasticsearch would ever perform hundreds of thousands of shard allocations like this.

I'm sorry, what I said here is wrong. We also create new instances of these things when the index settings or metadata are updated. Perhaps this is happening a lot?

If you grab some fresher heap dumps and they show the same pattern then I would look into some of those instances further: IndexMetaData.index.name is the String name of the index in question - perhaps this'll tell us something?