I am surprised that there is no config setting to limit the cache memory size of Lucene? This means Lucene will greedily eat up all memory and nobody can stop it?
Lucene just maps files into memory and has little control over how much of it is made part of the process's working set (i.e. the part of the process's address space that's in RAM). See Archived MSDN and TechNet Blogs | Microsoft Learn for some more background. It mentions a tool that can be used to limit the maximum working set size of a process—perhaps it's useful to you?