Logstash crashes with OOM

Elastic Stack Logstash
1

Logstash version - 6.2.0

logstash.yml has

pipeline.workers: 8
pipeline.batch.size: 250
pipleline.batch.delay: 5
log.level: warn
path.log : /usr/share/logstash

It is running in pod in kubernetes with min 3g and max 5g of memory. The pod crashes after 1 hour of usage and OOM error. While analyzing the heap found this.

</>
Problem Suspect 1

68 instances of "io.netty.util.concurrent.DefaultEventExecutor" , loaded by "org.jruby.util.JRubyClassLoader @ 0x6c3992e80" occupy 4,364,500,264 (83.46%) bytes.

Biggest instances:

  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3828eb0 - 112,381,328 (2.15%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f63c8 - 108,620,792 (2.08%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817e08 - 100,920,304 (1.93%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456138 - 100,211,048 (1.92%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456888 - 95,707,344 (1.83%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456820 - 91,473,176 (1.75%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817e70 - 89,591,048 (1.71%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d381a9e8 - 88,260,400 (1.69%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39a9538 - 88,139,568 (1.69%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817c68 - 85,195,688 (1.63%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f3578 - 83,102,248 (1.59%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456750 - 82,410,056 (1.58%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f3510 - 81,845,608 (1.56%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f35e0 - 81,734,736 (1.56%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3923900 - 81,014,544 (1.55%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d381a918 - 80,729,792 (1.54%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d38412d8 - 78,563,144 (1.50%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3840f98 - 77,632,384 (1.48%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39267b8 - 77,357,744 (1.48%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39a8ff0 - 75,636,776 (1.45%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d34566e8 - 75,467,808 (1.44%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39a8f20 - 75,290,592 (1.44%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39a8f88 - 75,134,256 (1.44%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d383e280 - 74,776,152 (1.43%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d34564e0 - 72,343,896 (1.38%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f6228 - 71,541,520 (1.37%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456478 - 71,489,152 (1.37%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f36b0 - 71,307,352 (1.36%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3844878 - 70,976,424 (1.36%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3828e48 - 70,949,224 (1.36%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3844810 - 70,415,992 (1.35%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d34565b0 - 69,949,520 (1.34%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d381a980 - 69,782,704 (1.33%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d381a8b0 - 69,389,880 (1.33%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3841000 - 66,237,624 (1.27%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3926750 - 65,789,608 (1.26%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d34561a0 - 65,720,384 (1.26%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39266e8 - 64,684,776 (1.24%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3828de0 - 63,547,248 (1.22%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817b30 - 63,459,752 (1.21%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39a90c0 - 61,840,136 (1.18%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3923968 - 61,683,728 (1.18%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456618 - 61,615,632 (1.18%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817d38 - 60,920,120 (1.16%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817cd0 - 60,482,864 (1.16%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f3648 - 59,936,400 (1.15%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d38449b0 - 58,924,864 (1.13%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d38448e0 - 57,296,960 (1.10%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d381a7e0 - 56,577,712 (1.08%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d39f6290 - 55,650,808 (1.06%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d34567b8 - 55,516,976 (1.06%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3456680 - 54,447,392 (1.04%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3844948 - 53,432,888 (1.02%) bytes.
  • io.netty.util.concurrent.DefaultEventExecutor @ 0x6d3817c00 - 52,301,264 (1.00%) bytes.
    </>
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.