Even if I disable machine learning plugin, ML is still running

before the deployment in aks, I created docker image of elk:8.3.3 with my configuration in alpine:3.15.0. when i tried to deploy to aks, I saw this error in the log.

org.elasticsearch.bootstrap.StartupException: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory

When I debugged the image in the docker. The file was exist. That made me a little confused.
How can i solve this? Thanks for help…Also I disabled the xpack.ml in the elasticsearch.yml.

Full stack trace depicted below.

warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [1151]
WARNING: Group of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [3151]
WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [1151]
WARNING: Group of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [3151]
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-17-openjdk/; using ES_JAVA_HOME
[2022-08-17T12:42:33,345][WARN ][o.e.c.l.LogConfigurator  ] [jocasta-elasticsearch-logging-2] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /usr/share/elasticsearch/config/log4j2.properties
[2022-08-17T12:42:33,431][ERROR][o.e.b.Elasticsearch      ] [jocasta-elasticsearch-logging-2] fatal exception while booting Elasticsearch
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:228) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) [elasticsearch-8.3.3.jar:?]
Caused by: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:171) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.3.jar:?]
	... 1 more
Caused by: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1143) ~[?:?]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
	at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:147) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:92) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.3.jar:?]
	... 1 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:314) ~[?:?]
	at java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[?:?]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
	at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:147) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:92) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224) ~[elasticsearch-8.3.3.jar:?]
	... 1 more
org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=2, No such file or directory
Likely root cause: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:147)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:92)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:224)
	at org.elasticsearch.server@8.3.3/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/kubernetes-logging.log
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/kubernetes-logging.log

ERROR: Elasticsearch exited unexpectedly

And this is my elasticsearch.yml :

cluster.name: kubernetes-logging

node.name: ${NODE_NAME}
#node.master: ${NODE_MASTER}
#node.data: ${NODE_DATA}

transport.port: ${TRANSPORT_PORT}
http.port: ${HTTP_PORT}

path.data: /data


node.store.allow_mmap: false


xpack.license.self_generated.type: basic


xpack.security.enabled: true
#xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.ml.enabled: false

xpack.security.transport.ssl.enabled: true
xpack.security.enrollment.enabled: false
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

network.host: 0.0.0.0
cluster.initial_master_nodes: [${NODE_NAME}]
http.host: [_local_, _site_]
transport.host: [_local_, _site_]

It seems you are attempting to run ES on alpine which is an unsupported Linux distribution. Also see: Elasticsearch failing to start due to X-Pack

1 Like

I know, the ticket that you mention. I think, I should give you more specific details. My elasticsearch:7.17.3 is running fine with the alpine image. My Docker file like that:

FROM alpine:3.15.0

RUN apk update
RUN apk add --no-cache openjdk8-jre
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk/

ENV VERSION 7.17.3
ENV DOWNLOAD_URL "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${VERSION}-linux-x86_64.tar.gz"
ENV AZURE_PLUGIN_DOWNLOAD_URL "https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-azure/repository-azure-${VERSION}.zip"

RUN apk add --upgrade --no-cache bash curl nghttp2 musl-utils ssl_client busybox apk-tools libjpeg-turbo libx11


# Download/Extract & Setup elasticsearch.tar.gz for the given version
WORKDIR /tmp
RUN mkdir -p /usr/share/elasticsearch
RUN mkdir -p es-download.tmp \
    && cd es-download.tmp \
    && apk add --no-cache -t tag-es-download wget \
    && echo "Downloading $DOWNLOAD_URL" \
    && wget -q -O elasticsearch.tar.gz "$DOWNLOAD_URL" \
    && echo "Extracting elasticsearch.tar.gz" \
    && tar -xf elasticsearch.tar.gz \
    && echo "Copying content to the /usr/share/elasticsearch" \
    && cp -r elasticsearch-$VERSION/* /usr/share/elasticsearch/ \
    && cd .. \
    && echo "Cleanup downloaded content & packages" \
    && rm -rf es-download.tmp \
    && apk del --purge tag-es-download

RUN wget -q -O repository-azure-${VERSION}.zip "$AZURE_PLUGIN_DOWNLOAD_URL" \
  && echo "0a82a4b591150f2604eaed94cac16dfd69892a0fb8b260fcd4f847d06e63de5ec539fb5960c605485c34c2eb1a2f650eca4e01c71789be77d75f094c7bfac2a2  repository-azure-${VERSION}.zip" | sha512sum -c

# remove x-pack plugins that we are not using even if these are disabled in elasticsearch.yaml
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-monitoring
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-graph
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-watcher
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-ml

WORKDIR /
VOLUME ["/data"]
EXPOSE 9200 9300

WORKDIR /usr/share/elasticsearch/
#hauhaf0: Health check needs authentication, that is why we need custom script
COPY docker-entrypoint.sh bin/
COPY health_probe.sh bin/
COPY run.sh bin/
COPY config/roles.yml config/elasticsearch.yml config/log4j2.properties config/

RUN chmod -R 777 config
RUN chmod -R 777 plugins
RUN chmod -R 777 /data
CMD ["bin/run.sh"]

I just updated the elk, kibana and java version (elk:8.3.3 java: openjdk17-jre)

Then configured my elasticsearch.yml like defined above and removed the lines below, because the x-pack-monitoring disable is deprecated in 8.3.3 version.

RUN rm -rf /usr/share/elasticsearch/modules/x-pack-monitoring
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-graph
RUN rm -rf /usr/share/elasticsearch/modules/x-pack-watcher

At this time, kibana looking for the ml files that makes me confused. Because i disabled in the elasticsearch.yml

OK, even if unsupported, the workaround of removing the x-pack-ml module should still work I would think.

This implies something goes wrong in your image when attempting to remove the x-pack-ml module. I think the key will be to first debug why the module directory is not being removed.

I think, kibana:8.3.3 caused the error. When I got in the pod, I caught that the ml folder was removed. And all pods worked normal.

I did port forward to check the elk, It was worked, as it should. Then I deployed kibana:8.3.3, the logs were getting weird:

[2022-08-17T18:35:14,119][INFO ][o.e.c.m.MetadataCreateIndexService] [jocasta-elasticsearch-logging-2] [.tasks] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2022-08-17T18:35:14,713][INFO ][o.e.c.r.a.AllocationService] [jocasta-elasticsearch-logging-2] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.tasks][0]]])." previous.health="YELLOW" reason="shards started [[.tasks][0]]"
[2022-08-17T18:35:17,738][WARN ][r.suppressed             ] [jocasta-elasticsearch-logging-2] path: /_xpack, params: {}
java.lang.IllegalStateException: failed to find action [ActionType [cluster:monitor/xpack/info/ml]] to execute
	at org.elasticsearch.client.internal.node.NodeClient.transportAction(NodeClient.java:156) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:115) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.action.TransportXPackInfoAction.doExecute(TransportXPackInfoAction.java:79) ~[?:?]
	at org.elasticsearch.xpack.core.action.TransportXPackInfoAction.doExecute(TransportXPackInfoAction.java:28) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:398) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:412) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:171) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:402) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:283) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
	at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.rest.action.RestXPackInfoAction.lambda$prepareRequest$0(RestXPackInfoAction.java:67) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:112) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:106) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:349) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:203) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:122) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:189) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:217) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:162) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:146) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:82) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:181) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:137) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100) [x-pack-security-8.3.3.jar:?]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345) [elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98) [transport-netty4-8.3.3.jar:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88) [transport-netty4-8.3.3.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.76.Final.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-17T18:35:17,819][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [jocasta-elasticsearch-logging-2] updating index lifecycle policy [.alerts-ilm-policy]
[2022-08-17T18:35:18,195][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [jocasta-elasticsearch-logging-2] updating index lifecycle policy [.preview.alerts-security.alerts-policy]

By the way, when I checked my current version of elk:7.17.3 upgrade assistant, Now, kibana:7.17.3 ml configuration is deprecated and it is critical for my configuration. Thanks for the advice, I changed the image as ubuntu:18.04 and it worked for me. (2 days ago, the upgrade assistant didnt fınd anything critical for kibana :smile: )

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.