Script groovy may cause memory leak in elastic v5.4

[1222kkk (new to Es)]

Recently, I found my elasticsearch cluster always get full gc every two days.

Cluster settings are as follows:
es version: v2.3.2
2 * node, every node set 8G for es
data size is just 500MB+ with 5 indices

I recognized that groovy memory leak issue may exist in v2.3.2.

Groovy scripting - possible memory leak ES 2.3.3

So I try to reduce the use of script in update and user params instead if script lines have dynamic params.

Everything seems to goes well before jstat -gcold prints as follows:

I think full gc will finally comes since the use of old gc still increase.

So I try to update es version to v5.4.x. One days later, org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry still increase as v2.3.2.

num     #instances         #bytes  class name
----------------------------------------------
   1:        459350      992735952  [B
   2:        547852      219588688  [C
   3:       1826845      102303320  org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry
   4:       1485050       62888720  [Ljava.lang.Object;
   5:        998557       31953824  java.util.HashMap$Node
   6:       1322667       31744008  org.codehaus.groovy.util.FastArray
   7:        146991       21167600  [Lorg.codehaus.groovy.util.ComplexKeyHashMap$Entry;
   8:        113728       14367840  [Ljava.util.HashMap$Node;
   9:        421638       13492416  org.wltea.analyzer.dic.DictSegment
  10:        529427       12706248  java.lang.String

Here is my question:

groovy memory leak is solved in es v2.4.0, why it seems doesn't work...
Is there any best way to use script in update which may not result in cluster frequent full gc ?

Thank you!

[1222kkk (new to Es)]

anyone face the same problem or any suggestions ?

[1222kkk (new to Es)]

jmap -histo:live

 num     #instances         #bytes  class name
----------------------------------------------
   1:       7023686      393326416  org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry
   2:       5403346      152118584  [Ljava.lang.Object;
   3:       5085873      122060952  org.codehaus.groovy.util.FastArray
   4:       2831417       90605344  java.util.HashMap$Node
   5:        565199       81389552  [Lorg.codehaus.groovy.util.ComplexKeyHashMap$Entry;
   6:         90390       78825856  [B
   7:        591953       66105088  [C
   8:       1332338       42634816  org.codehaus.groovy.util.SingleKeyHashMap$Entry
   9:         40378       41983392  [Lorg.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry;
  10:        807474       38758752  java.lang.invoke.MethodHandleImpl$CountingWrapper
  11:        181720       32837448  [Ljava.util.HashMap$Node;
  12:        686366       27454640  java.lang.invoke.BoundMethodHandle$Species_LL
  13:        687156       21988992  java.util.concurrent.ConcurrentHashMap$Node
  14:        243943       21466984  java.lang.reflect.Method
  15:        417310       20199984  [I
  16:        403742       19379616  java.lang.invoke.BoundMethodHandle$Species_L4
  17:        605507       19376224  groovy.lang.MetaBeanProperty
  18:        397889       19098672  java.util.HashMap
  19:        203232       16258560  java.lang.reflect.Constructor
  20:        583558       14005392  java.lang.String

[rjernst (Ryan Ernst)]

I recommend you switch to using painless scripts (added in 5.0). It is unlikely anyone will look at debugging a groovy issue as it was removed in 6.0.

[1222kkk (new to Es)]

Fine, I will try it.
Now I temporarily solved the issue by using all params in label params instead of using them directly in script line and it worked.