Hi,
We have existing elasticsearch cluster(6 nodes) running on GKE. I am trying to create backup repository for creating backup snapshot https://www.elastic.co/guide/en/elasticsearch/plugins/7.8/repository-gcs.html
We use this script for kubernetes pod startup. I have added plugin installation command to my existing startup script and restarted all of my pods, plugin got installed but pods failed with this error log.
elastic_entry.sh
#! /usr/bin/env bash
# Setting memlock
cat >> /etc/security/limits.conf << __LIMITS_CONF__
elasticsearch hard memlock unlimited
elasticsearch soft memlock unlimited
elasticsearch - nofile 65536
__LIMITS_CONF__
# Changing config permission. The script assume elasticsearch.yml has been mounted as secret volume
cp /tmp/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
cp /tmp/service-account.json /usr/share/elasticsearch/config/service-account.json
chown elasticsearch:elasticsearch /usr/share/elasticsearch/config/elasticsearch.yml
chmod 400 /usr/share/elasticsearch/config/elasticsearch.yml
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config
chmod 777 /usr/share/elasticsearch/config/service-account.json
ls -lRt /usr/share/elasticsearch/config/
# Changing data volume permission
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
#Install plugin
cd /usr/share/elasticsearch/bin
elasticsearch-plugin install --batch repository-gcs
cd /usr/share/elasticsearch/
# Virtual memory mapping
sysctl -w vm.max_map_count=262144
su -c "/usr/share/elasticsearch/bin/es-docker" -s /bin/bash elasticsearch
Error Log
2020-08-18 16:54:00.055 IST
-> Installed repository-gcs
Info
2020-08-18 16:54:00.130 IST
vm.max_map_count = 262144
Info
2020-08-18 16:54:01.785 IST
2020-08-18 11:24:01,782 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register") at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) at java.lang.SecurityManager.checkPermission(SecurityManager.java:585) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.logging.log4j.core.jmx.Server.register(Server.java:389) at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167) at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:556) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:261) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:206) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:221) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:197) at org.elasticsearch.common.logging.LogConfigurator.configureStatusLogger(LogConfigurator.java:175) at org.elasticsearch.common.logging.LogConfigurator.configureWithoutConfig(LogConfigurator.java:99) at org.elasticsearch.cli.Command.main(Command.java:85) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Info
2020-08-18 16:54:01.785 IST
Error
2020-08-18 16:54:02.439 IST
Exception in thread "main" java.lang.IllegalStateException: status logger logged an error before logging was configured at org.elasticsearch.common.logging.LogConfigurator.checkErrorListener(LogConfigurator.java:128) at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:117) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:316) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) at org.elasticsearch.cli.Command.main(Command.java:88) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Error
2020-08-18 16:54:02.439 IST
Refer to the log for complete error details.