Kibana Extremely slow Start Up Time on Openshift/Kubernetes while Optimizing and caching bundles

Elastic Stack Kibana
1

Hi guys,

The deployment of kibana on openshift is extremely slow after or during this step:

"Optimizing and caching bundles.... This may take a few minutes"

The deployment takes about 7-8 minutes. You have an Idea how I can increase the kibana start up time (maybe relocate some configurations or files to persistent storage)?

Thank you

2

Hello,

Which version of kibana are you running? Are you building your own image?

3

I run version 7.6.2. I use the image provided by elasticsearch. I use a different base image (ubi8), due to security restrictions in openshift

edit: As ressources I use 2 cores and 4000Mi, which should be enough i guess

4

The optimize is being triggered due to a configuration change, usually a plugin added or disabled. Starting in 7.10 this process is removed entirely, but for now you can call bin/kibana --optimize during the creation of your image so it doesn't have to be done at runtime.

5

I'll give it a try and report. Thank you

6

When I try to execute the ./bin/kibana --optimize i receive following error:

FATAL Error: EACCES: permission denied, unlink /usr/share/kibana/optimize/bundles/uptime.style.css'

Do you have any idea wher ein the Dockerfile i can execute the optimization process ?

7

Can you provide the Dockerfile you have?

This might also help: How to do custom docker image that has certain features disabled but run bin/kibana --optimize instead of installing the plugin.

8

Basically this is my Dockerfile https://github.com/elastic/dockerfiles/blob/7.6/kibana/Dockerfile, accept I use a different base image from Redhat (ubi8) and my entrypoint is:

ENTRYPOINT [ "sh", "-c", "kibana" ]

and not

ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]

I tried to add

RUN ./bin/kibana --optimize in line 60 but this leads to:

FATAL Error: EACCES: permission denied, unlink /usr/share/kibana/optimize/bundles/uptime.style.css'

9

As a heads up, we will be shipping a docker image based on UBI 8.2 starting in 7.10. https://github.com/elastic/kibana/pull/74656

When you are running the container, what configuration are you using? It's important to identify what is causing the optimize as you will need to have that setting present when you optimize, otherwise it will still do that when you startup the resutling container.

10

You mean kibana configuration or what configuration you mean exactly?

Kibana config:

kibanaConfig:
  kibana.yml: |
    server.name: kibana01
    server.host: "0.0.0.0"
    elasticsearch.hosts: "https://elastic.url:9200"
    elasticsearch.username: ${ELASTICSEARCH_USERNAME}
    elasticsearch.password: ${ELASTICSEARCH_PASSWORD}
    xpack.security.enabled: true
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-root-ca
      verificationMode: full
11

Yeah, that is the configuration I was referring to but I am not seeing anything there which would trigger a re-optimize.

Is there any environment variables you are passing to the container?

12

Yes, I do. And one of them is:

NODE_OPTIONS = --max-old-space-size=1800

That looks pretty small. Maybe I should try to increase this one

All others are some secerts, elastic host url and kibana host

Edit: That did not help

Edit2:

I have added now:

RUN chown -R kibana:kibana /opt/kibana/optimize
RUN chown -R kibana:kibana /usr/share/kibana/optimize

to my Dockerfile.

Now the optimization process sucessfuly done within the build process. Great success.

But now when the container is starting I have again same errors:

Babel could not write cache to file: /usr/share/kibana/optimize/.babel_register_cache.json 
due to a permission issue. Cache is disabled.

and

FATAL Error: EACCES: permission denied, unlink '/usr/share/kibana/optimize/bundles/uptime.style.css'

13

Now I managed to do the optimization in the build process and get rid of the permission errors during the build and in the deployment.

But kibana still does the optimization process in the deployment again. So running ./bin/kibana --optimize had no affect on the deployment of the application!

14

Here is what I have for my Dockerfile - I am using UBI, changed to microdnf

#
# ** THIS IS AN AUTO-GENERATED FILE **
#

################################################################################
# Build stage 0
# Extract Kibana and make various file manipulations.
################################################################################
FROM registry.access.redhat.com/ubi8/ubi-minimal:latest AS prep_files
# Add tar and gzip
RUN microdnf update -y && microdnf install -y tar gzip findutils && microdnf clean all
RUN cd /opt && curl --retry 8 -s -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.6.2-linux-x86_64.tar.gz && cd -
RUN mkdir /usr/share/kibana
WORKDIR /usr/share/kibana
RUN tar --strip-components=1 -zxf /opt/kibana-7.6.2-linux-x86_64.tar.gz
# Ensure that group permissions are the same as user permissions.
# This will help when relying on GID-0 to run Kibana, rather than UID-1000.
# OpenShift does this, for example.
# REF: https://docs.openshift.org/latest/creating_images/guidelines.html
RUN chmod -R g=u /usr/share/kibana
RUN find /usr/share/kibana -type d -exec chmod g+s {} \;

################################################################################
# Build stage 1
# Copy prepared files from the previous stage and complete the image.
################################################################################
FROM registry.access.redhat.com/ubi8/ubi-minimal:latest
EXPOSE 5601

# Add Reporting dependencies.
RUN microdnf update -y && microdnf install -y fontconfig freetype shadow-utils findutils && microdnf clean all

# Add an init process, check the checksum to make sure it's a match
RUN curl -L -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_amd64
RUN echo "37f2c1f0372a45554f1b89924fbb134fc24c3756efaedf11e07f599494e0eff9  /usr/local/bin/dumb-init" | sha256sum -c -
RUN chmod +x /usr/local/bin/dumb-init


# Bring in Kibana from the initial stage.
COPY --from=prep_files --chown=1000:0 /usr/share/kibana /usr/share/kibana
WORKDIR /usr/share/kibana
RUN ln -s /usr/share/kibana /opt/kibana

ENV ELASTIC_CONTAINER true
ENV PATH=/usr/share/kibana/bin:$PATH

# Set some Kibana configuration defaults.
COPY --chown=1000:0 config/kibana.yml /usr/share/kibana/config/kibana.yml

# Add the launcher/wrapper script. It knows how to interpret environment
# variables and translate them to Kibana CLI options.
COPY --chown=1000:0 bin/kibana-docker /usr/local/bin/

# Ensure gid 0 write permissions for OpenShift.
RUN chmod g+ws /usr/share/kibana && find /usr/share/kibana -gid 0 -and -not -perm /g+w -exec chmod g+w {} \;

# Provide a non-root user to run the process.
RUN groupadd --gid 1000 kibana && useradd --uid 1000 --gid 1000 --home-dir /usr/share/kibana --no-create-home kibana
USER kibana

LABEL org.label-schema.schema-version="1.0" org.label-schema.vendor="Elastic" org.label-schema.name="kibana" org.label-schema.version="7.6.2" org.label-schema.url="https://www.elastic.co/products/kibana" org.label-schema.vcs-url="https://github.com/elastic/kibana" org.label-schema.license="Elastic License" org.label-schema.usage="https://www.elastic.co/guide/en/kibana/index.html" org.label-schema.build-date="2020-03-26T07:47:43.654Z" license="Elastic License"

ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]

CMD ["/usr/local/bin/kibana-docker"]

Built it: docker build --tag discuss-245203:7.6.2 .

Ran it: docker run --rm --network host -v "/home/tyler/elastic/dockerfiles/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml" discuss-245203:7.6.2

Here is what I have for the kibana.yml which is based off yours:

server.name: kibana01
server.host: "0.0.0.0"
elasticsearch.hosts: "http://127.0.0.1:9200"
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.security.enabled: true
xpack.security.encryptionKey: foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo

With this I have verified that it's not running the optimize step. I really think either the optimize directory is getting removed, or a plugin is being disabled, etc. Something is being missed.

Can you provide reproduction steps to reproduce?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.