Using LDAP to set up Users and Roles in elasticsearch.yml file

Elastic Stack Elasticsearch
1

I am trying to set up user/role details in the elasticsearch.yaml file with LDAP settings, and need some support. I have followed elastic documentation, but it’s not working. I have added below details (with our own details) to every node in the elasticsearch.yaml file and have created a role_mappings.yml file that I added to the repo as well. I am getting “java.lang.IllegalArgumentException: unknown setting [xpack.security.authc.ldap.ldap1.order] please check that any required plugins are installed.” The goal is to be able ot have user and role values set up in the repo. Is this possible? Any help is appreciated!
What I added to elasticsearch.yml:

       ` ldap:
          ldap1:
              order: 0
              url: "ldap://redacted"
              user_dn_templates:
                - "cn={redacted}, ou=users, o=engineering, dc=bhg, dc=com"
              group_search:
                base_dn: "dc=redacted,dc=com"
                files:
                  role_mapping: "/mnt/elasticsearch/role_mapping.yml"
                unmapped_groups_as_roles: false`

role_mapping.yml file I added:

`"super_user:

  • "cn=admins,cn=users,dc={redacted},dc=com"
    user:
  • "cn=users,dc={redacted},dc=com"
  • "cn=admins,dc={redacted},dc=com"
  • "cn={redacted},cn=admins,dc={redacted},dc=com"
  • "cn=test,cn=users,dc={redacted},dc=com"`
2

Can you share your entire elasticsearch.yml?

It seems that there is something wrong in your config.

java.lang.IllegalArgumentException: unknown setting [xpack.security.authc.ldap.ldap1.order]

Did you define ldap under the realms as explained in the documentation?

3

Thanks for the fast response! I can't share my entire file as its for an org, but I have added this to each node in my elasticsearch.yml file:

nodeSets:
  - name: master
    count: {{ .Values.masterCount }}
    config:
      node.roles: ["master"]
      xpack.security.authc.token.enabled: true
      xpack.security.audit.enabled: false
      xpack:
        security:
          authc:
            realms:
            {{- if .Values.ldap}}
            ldap1:
              type: ldap
              order: 0
              url: "ldap://{redacted}"
              user_dn_templates:
                - "cn={sswilley}, ou=users, o=engineering, dc={redacted}, dc=com"
              group_search:
                base_dn: "dc={redacted},dc=com"
              files:
                role_mapping: "/mnt/elasticsearch/role_mapping.yml"
              unmapped_groups_as_roles: false
            {{- end }}

I took out the extra ldap, left ldap1 and added type :ldap
I have added this per the elastic documentation here: elastic documentation

4

Still getting this error, although it looks it wont read the oidc settings after the LDAP elastic settings. In order to set up a realm, I am only adding the below change, right?

realms: ldap1: type: ldap
Error:
"error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [xpack.security.authc.ldap1.oidc.cloud-oidc.op.authorization_endpoint] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"

5

The indentation seems to be wrong, compare with the example on the documentation.

xpack:
  security:
    authc:
      realms:
        ldap1:
          type: ldap

In your config ldap1 is on the same column as realms, this may be the issue.

6

Ok, so I updated my file to look like the below, but now I am getting this error:

"log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev","error.type":"java.lang.IllegalStateException","error.message":"failed to load plugin class [org.elasticsearch.xpack.security.Security]"

 - name: master
    count: {{ .Values.masterCount }}
    config:
      node.roles: ["master"]
      xpack.security.authc.token.enabled: true
      xpack.security.audit.enabled: false
      xpack:
        security:
          authc:
            realms:
            {{- if .Values.ldap }}
              ldap1:
                type: ldap
                order: 0
                url: "ldap://{redacted}"
                user_dn_templates:
                  - "cn={sswilley}, ou=users, o=engineering, dc={redacted}, dc=com"
                group_search:
                  base_dn: "dc={redacted},dc=com"
                files:
                  role_mapping: "/mnt/elasticsearch/role_mapping.yml"
                unmapped_groups_as_roles: false
            {{- end }}
7

Here are the complete logs:

{"@timestamp":"2023-01-18T20:27:20.400Z", "log.level": "INFO", "message":"version[8.5.3], pid[69], build[docker/4ed5ee9afac63de92ec98f404ccbed7d3ba9584e/2022-12-05T18:22:22.226119656Z], OS[Linux/5.4.0-1090-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-18T20:27:20.408Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-18T20:27:20.408Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1285768002529455253, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8g, -Xmx8g, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-18T20:27:23.354Z", "log.level": "INFO", "message":"Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: Troubleshoot dependency version conflicts when using the Azure SDK for Java | Microsoft Learn", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev"}
{"timestamp": "2023-01-18T20:27:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"@timestamp":"2023-01-18T20:27:24.504Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"eck-dev-es-coordinator-0","elasticsearch.cluster.name":"eck-dev","error.type":"java.lang.IllegalStateException","error.message":"failed to load plugin class [org.elasticsearch.xpack.security.Security]","error.stack_trace":"java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.security.Security]\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:482)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:290)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.(PluginsService.java:159)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:634)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.(Node.java:411)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.(Node.java:318)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:600)\n\t... 9 more\nCaused by: java.lang.IllegalArgumentException: Incorrect realm settings found. Realm settings have been changed to include the type as part of the setting key.\nFor example 'xpack.security.authc.realms.file.my_file.order'\nFound invalid config: xpack.security.authc.realms.ldap1.type, xpack.security.authc.realms.ldap1.order, xpack.security.authc.realms.ldap1.user_dn_templates, xpack.security.authc.realms.ldap1.url, xpack.security.authc.realms.ldap1.unmapped_groups_as_roles\nPlease see the breaking changes documentation.\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.validateRealmSettings(Security.java:1391)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.runStartupChecks(Security.java:529)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.(Security.java:520)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.(Security.java:509)\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)\n\t... 12 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/eck-dev.log

ERROR: Elasticsearch exited unexpectedly

8

Just saw it now, you are using a old config format.

There is no type: ldap since version 7.X, the following line from your error log tells you that.

Caused by: java.lang.IllegalArgumentException: Incorrect realm settings found. Realm settings have been changed to include the type as part of the setting key.\nFor example 'xpack.security.authc.realms.file.my_file.order'\nFound invalid config: xpack.security.authc.realms.ldap1.type, xpack.security.authc.realms.ldap1.order, xpack.security.authc.realms.ldap1.user_dn_templates, xpack.security.authc.realms.ldap1.url, xpack.security.authc.realms.ldap1.unmapped_groups_as_roles\nPlease see the breaking changes documentation.\n\

Since you are running version 8.5.3, you need to follow the example in the documentation for this version.

It needs to be something like this:

      xpack:
        security:
          authc:
            realms:
              ldap:
                ldap1:
                  order: 0
                  url: "ldap://{redacted}"
                  user_dn_templates:
                    - "cn={sswilley}, ou=users, o=engineering, dc={redacted}, dc=com"
                  group_search:
                    base_dn: "dc={redacted},dc=com"
                  files:
                    role_mapping: "/mnt/elasticsearch/role_mapping.yml"
                  unmapped_groups_as_roles: false
9

Thanks for the details! Posting the logs below. It looks like I am getting some security issue with my roel_mapping.yml file.

"1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"timestamp": "2023-01-19T15:16:00+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"@timestamp":"2023-01-19T15:16:00.869Z", "log.level": "INFO", "message":"loaded module [aggs-matrix-stats]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.869Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.870Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.870Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.870Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.871Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.871Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.871Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.871Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.873Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.873Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.873Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.873Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.874Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.874Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.877Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.878Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.878Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.879Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.879Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.880Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.880Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.881Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.881Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.882Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.882Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.883Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.883Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.884Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.884Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.885Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.885Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.886Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.886Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.887Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.887Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.888Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.888Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.889Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.889Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.890Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.890Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.890Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.891Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.891Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.894Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.895Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.895Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.895Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.896Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.896Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.896Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.897Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.897Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.897Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.897Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.898Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.898Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.899Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.899Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.899Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.900Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.900Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.900Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.900Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:00.901Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"timestamp": "2023-01-19T15:16:05+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"@timestamp":"2023-01-19T15:16:05.198Z", "log.level": "WARN", "message":"Jan 19, 2023 3:16:05 PM org.apache.lucene.store.MMapDirectory lookupProvider", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:05.199Z", "log.level": "WARN", "message":"WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:05.213Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdd)]], net usable_space [9.7gb], net total_space [9.7gb], types [ext4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:05.214Z", "log.level": "INFO", "message":"heap size [8gb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:05.277Z", "log.level": "INFO", "message":"node name [eck-dev-es-coordinator-1], node ID [ATxkWnqjSKOyLv7MUPSEbA], cluster name [eck-dev], roles ", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:09.723Z", "log.level": "INFO", "message":"Security is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev"}
{"@timestamp":"2023-01-19T15:16:09.746Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"eck-dev-es-coordinator-1","elasticsearch.cluster.name":"eck-dev","error.type":"java.lang.IllegalStateException","error.message":"security initialization failed","error.stack_trace":"java.lang.IllegalStateException: security initialization failed\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:577)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.lambda$new$16(Node.java:709)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.(Node.java:724)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.(Node.java:318)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/mnt/elasticsearch/role_mapping.yml" "read")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1068)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:751)\n\tat java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:780)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:537)\n\tat java.base/java.nio.file.Files.exists(Files.java:2521)\n\tat org.elasticsearch.xcore@8.5.3/org.elasticsearch.xpack.core.XPackPlugin.resolveConfigFile(XPackPlugin.java:401)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.support.DnRoleMapper.resolveFile(DnRoleMapper.java:87)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.support.RoleMappingFileBootstrapCheck.create(RoleMappingFileBootstrapCheck.java:48)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)\n\tat java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.InternalRealms.getBootstrapChecks(InternalRealms.java:187)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:612)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:565)\n\t... 17 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/eck-dev.log
{"timestamp": "2023-01-19T15:16:10+00:00", "message": "readiness probe failed", "curl_rc": "7"}

ERROR: Elasticsearch exited unexpectedly

10

Yes, you need to fix the permissions, the user running elasticsearch needs to be able to read that file.

Also, when sharing logs please use the preformatted text option, the </> buton, or put your log between triple single quotes use ``` one line above and one below, it makes way easier to read.

11

Will do next time! So, I am very new to ES. I am not sure how to fix the permissions. I have been updating our .yml files through Azure DevOps repo. Are you talking about the way that role_mappings.yml file is mounted to the elasticsearch.yml file? I am not sure how to mount it. I added the below code from the elastic documentation, and created a role_mapping.yml file. Is that correct?

role_mapping: "/mnt/elasticsearch/role_mapping.yml"

role_mapping.yml file I added:

super_user:
  - "cn=admins,cn=users,dc={redacted},dc=com"
user:
  - "cn=test,cn=users,dc={redacted},dc=com"

This is the structure of our repo:

image
image293×746 6.31 KB

Thanks in advance.

13

Is there any update on this? I have mounted the role_mapping.yml file the way the documentation lists.

14

I do not use ECK, so I can not help further, but you have a permissions issue, which is not related to elasticsearch per se, but to the way you are running it.

Which documentation did you follow? Please share the link, maybe someone else can give more insight about it, also double check all the steps to see if something was missing in your configuration.

Do you have a paid license or are you using the trial? If you have a paid license you may reach the support team for better help.

Also, keep in mind that there is no SLA in this forum.

15

Its a paid license.

The documentation I followed is this for LDAP configs
And this for role mapping

Ok thank you

16

Those documentations are about the configuration of the LDAP and the Role Mapping, your current issue is not related to this, is related to how you are running Elasticsearch.

It is a permission issue, which is a infrastructure issue, Elasticsearch can not read the file for your role mapping configuration, you need to fix this permission issue first.

Since you have a paid license, it is easier to open a ticket with the support team.

17

Ok thank you for your help!

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.