we use elasticsearch for the classic ELK stack. We're indexing on average about 18 million docs (log file entries) per day. ES is doing a fantastic job, but I was wondering about memory usage. It seems it is constantly increasing. We've only run this stack for about a month, so haven't expired (removed) any old data yet.
We're at about 600M docs now, 375G index size. Query speed is not important for us. ES is running at 5G heap size, constantly increasing (upped from initially 3G). Here are some graphs. It seems quite clear that memory usage is proportional to document count. Now I have read a lot about memory usage, and lots of information out there is for old versions of ES, but I've read about the "circuit breakers" and I had the impression that somehow it's possible to limit ES memory usage and force it to read data from disk if RAM becomes low.
I don't know if I need to change any indexing params. We have unique strings for session IDs, so those will definitely keep piling up. Are there any definite guides to make ES limited in memory usage if I don't care about query speed? Or is it unavoidable?
Here are the graphs (starting halfway through us starting to use ELK): http://imgur.com/a/6HaRZ
Thanks for all hints,