Unable to load config file. access denied (/usr/share/elasticsearch/.aws/config)

I'm setting elk stack with docker-compose. But I'm getting a warning. Does anyone know the cause or can guide me on what wrong step I took?

Warning from Terminal
{"@timestamp":"2022-06-19T10:40:48.631Z", "log.level": "WARN", "message":"Unable to load config file null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1068)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:751)\n\tat java.base/java.io.File.exists(File.java:829)\n\tat com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36)\n\tat com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41)\n\tat com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfilesConfigFile(BasicProfileConfigFileLoader.java:69)\n\tat com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfile(BasicProfileConfigFileLoader.java:55)\n\tat com.amazonaws.retry.internal.RetryModeResolver.profile(RetryModeResolver.java:92)\n\tat com.amazonaws.retry.internal.RetryModeResolver.resolveRetryMode(RetryModeResolver.java:83)\n\tat com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:46)\n\tat com.amazonaws.retry.RetryPolicy.<clinit>(RetryPolicy.java:35)\n\tat com.amazonaws.retry.PredefinedRetryPolicies.<clinit>(PredefinedRetryPolicies.java:30)\n\tat com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:89)\n\tat java.base/java.lang.Class.forName0(Native Method)\n\tat java.base/java.lang.Class.forName(Class.java:383)\n\tat java.base/java.lang.Class.forName(Class.java:376)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:59)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.<clinit>(S3RepositoryPlugin.java:53)\n\tat java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)\n\tat java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)\n\tat java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)\n\tat java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)\n\tat java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:236)\n\tat java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:546)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:496)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)\n\tat org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:757)\n\tat org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:704)\n\tat org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:487)\n\tat org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165)\n\tat org.elasticsearch.node.Node.<init>(Node.java:393)\n\tat org.elasticsearch.node.Node.<init>(Node.java:291)\n\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)\n\tat org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\n"}
My Docker-Compose file
version: "3.9"

services:
  setup:
    build:
      context: elasticsearch
      dockerfile: Dockerfile
    image: setup:local
    container_name: elk-setup
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    build:
      context: elasticsearch
      dockerfile: Dockerfile
    image: es01:local
    container_name: es01
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - logger.level=WARN
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    build:
      context: elasticsearch
      dockerfile: Dockerfile
    image: es02:local
    container_name: es02
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - logger.level=WARN
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    build:
      context: elasticsearch
      dockerfile: Dockerfile
    image: es03:local
    container_name: es03
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - logger.level=WARN
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    build:
      context: kibana
      dockerfile: Dockerfile
    image: kibana:local
    container_name: kibana
    volumes:
#      - ./kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
      - ./certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - xpack.security.encryptionKey=5h1e2yc1ogzv85q3ontrtmkjs366alyh
      - xpack.reporting.encryptionKey=ym1et5pob2xvohjzddc4y4bpg2235vxu
      - xpack.encryptedSavedObjects.encryptionKey=ck5j6uc335mfnkmpr8bdd5jon9fmjnx1
      - xpack.reporting.roles.enabled=false
      - logging.root.level=warn
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local

Hello @wedas

Welcome to elastic community :slight_smile: !!!

The mentioned error clearly states that there is file permission issue in configuration file in this directory /usr/share/Elasticsearch/.aws/config

The current permission of the configuration file is read . Hence change it to read and execute that would solve your issue.

Note: Apply the permission to folder and its subfolder and files recursively using -R .
Example: chmod -R 764 /usr/share/elasticsearch/.aws/config/*

Keep Posted !!! Thanks !!!

@sudhagar_ramesh But I'm not using aws or any of its products. Then why even Elasticsearch is looking for the aws credential/config files.

and there is no such directory .aws directory in my Elasticsearch container. So the above permission command will give an error.

Hello @wedas

In the logs, there is AccessControlException in AWS S3 repository.

Could you please check whether your elastic cluster or your servers is using any of AWS products.

That's the thing I'm saying l. I'm not using any AWS product. That's why I pasted docker-compose. You can look into that. I'm not using any of AWS product.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.