Kibana 7.6 crashing in docker

Hi,

While trying to run kibana 7.6.2 in docker I am constantly facing the below issue where kibana crashes. I am running this inside HahsiCorp Nomad.

<--- Last few GCs --->

[6:0x41307d0]   184257 ms: Mark-sweep 1020.5 (1079.9) -> 1020.3 (1075.9) MB, 1177.0 / 0.0 ms  (average mu = 0.073, current mu = 0.002) allocation failure GC in old space requested
[6:0x41307d0]   185516 ms: Mark-sweep 1020.3 (1075.9) -> 1020.1 (1073.9) MB, 1256.6 / 0.0 ms  (average mu = 0.038, current mu = 0.002) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x37149b75be1d]
    1: StubFrame [pc: 0x37149b75d1df]
Security context: 0x29568329e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x2978dbdb989] [/usr/share/kibana/node_modules/terser/dist/bundle.min.js:~1] [pc=0x37149c772fc7](this=0x320408f826f1 <undefined>)
    3: arguments adaptor frame: 1->0
    4: /* anonymous */(aka /* anonymous */) [0x2978dbdece9] [/usr/share/kibana/node_modules/terser/dist/bundle...

My configurations are below.

        image   = "kibana:7.6.2"
        command = "/usr/local/bin/kibana-docker"

        args = [
          "--elasticsearch.hosts=http://es7.service.consul:9200",
          "--server.host=0.0.0.0",
          "--server.name=es-kibana",
          "--server.port=5601",
          "--path.data=/alloc/data",
          "--elasticsearch.preserveHost=false",
          "--xpack.apm.ui.enabled=false",
          "--xpack.graph.enabled=false",
          "--xpack.ml.enabled=false",
          "--xpack.monitoring.enabled=false",
          "--xpack.security.enabled=false",
          "--xpack.reporting.enabled=false",
          "--xpack.watcher.enabled=false",
          "--xpack.monitoring.ui.container.elasticsearch.enabled=false",
        ]


      env {
        ES_JAVA_OPTS="-Xms1024m -Xmx1024m"
        NODE_OPTIONS="--max-old-space-size=1048"
      }

I have set the container memory limit to 2GB. I thought it might be issue with insufficient memory. But I see the same issue when I increase the memory as well.

So my questions are

  1. If this is really a memory issue, what is the min memory needed to run Kibana?
  2. Any issues with the options that used which might trigger this?

Any help is appreciated. Thanks in advance.

Thanks

Best Regards,
Vishwanath

Hi @vbhat,

I am thinking the NODE_OPTIONS heap size may be causing your error. Can you try to increase it?

Thanks,
Liza

Thanks for the reply @LizaD

When I had not set the NODE_OPTIONS, container exited with OOMKill almost every time (Sometimes it got stuck in optimising thing until health check timeout). That's why I reduced it, thinking it might help.

This brings me back to my first question. What is the right amount for NODE_OPTIONS and what minimum amount of memory needed to run Kibana? For now I'm trying with 2gb in staging environment.

Thanks

Best Regards,
Vishwanath

Thanks @vbhat, for Kibana memory minimum 1GB should be fine, if you plan to use reporting 2GB is minimum.

You should not have to set NODE_OPTIONS, so it might be good to get a full log of the errors without it set, but you can try NODE_OPTIONS="--max-old-space-size=4096" see if this works.

For Elasticsearch I found this document: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

Let us know,

Thanks,
Liza

Hi,

Thanks again for reply.

I was somehow able to run Kibana with trial and error method. It started running with below config

        image   = "kibana:7.6.2"
        command = "/usr/local/bin/kibana-docker"

        args = [
          "--elasticsearch.hosts=http://elastic.service.consul:9200",
          "--server.host=0.0.0.0",
          "--server.name=es-kibana",
          "--server.port=${NOMAD_PORT_http}",
          "--path.data=/alloc/data",
#          "--elasticsearch.preserveHost=false",
#          "--xpack.apm.ui.enabled=false",
#          "--xpack.graph.enabled=false",
#          "--xpack.ml.enabled=false",
#          "--xpack.monitoring.enabled=false",
#          "--xpack.security.enabled=false",
#          "--xpack.reporting.enabled=false",
#          "--xpack.watcher.enabled=false",
          "--xpack.monitoring.ui.container.elasticsearch.enabled=false",
        ]

      }

      env {
        ES_JAVA_OPTS="-Xms1024m -Xmx1024m"
      }

When I tried to run without NODE_OPTIONS, it was getting OOM killed because the I had set 2GB hard limit on memory. But I was able to run without xpack options and with elasticsearch.preserveHost=false.

Do you think there is anything in those commented out options that might lead to Kibana using more memory?

Thanks for your help again.

Thanks

Best Regards,
Vishwanath

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.