Received plaintext http traffic on an https channel

Cruz · January 26, 2023, 10:08pm

Is it normal for this to appear as a warning in the elasticsearch logs?
The first time I installed elasticsearch, this is what I saw in the elasticsearch logs.

[2023-01-27T06:18:35,815][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [master-node] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.16.7.75:9200, remoteAddress=/192.168.111.25:61483}
[2023-01-27T06:18:36,847][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [master-node] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.16.7.75:9200, remoteAddress=/192.168.111.25:61484}

What should I do to get rid of this?

I tried to get clear cache of my browser, I thought because I was accessing it using http not https that's why it appears like this but its still like that.

I am using Linux server 22.04 and elasticsearch version 8.6

When I set xpack.security.http.ssl and xpack.security.transport.ssl to false:

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

I got this error:
My elasticsearch seems not working

NFO ][o.e.i.r.RecoverySettings ] [master-node] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-01-27T05:46:42,469][INFO ][o.e.d.DiscoveryModule    ] [master-node] using discovery type [multi-node] and seed hosts providers [settings]
[2023-01-27T05:46:43,768][INFO ][o.e.n.Node               ] [master-node] initialized
[2023-01-27T05:46:43,769][INFO ][o.e.n.Node               ] [master-node] starting ...
[2023-01-27T05:46:43,783][INFO ][o.e.x.s.c.f.PersistentCache] [master-node] persistent cache index loaded
[2023-01-27T05:46:43,783][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [master-node] deprecation component started
[2023-01-27T05:46:43,874][INFO ][o.e.t.TransportService   ] [master-node] publish_address {172.16.7.75:9300}, bound_addresses {[::]:9300}
[2023-01-27T05:46:44,296][INFO ][o.e.b.BootstrapChecks    ] [master-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-27T05:46:44,337][INFO ][o.e.n.Node               ] [master-node] stopping ...
[2023-01-27T05:46:44,389][INFO ][o.e.n.Node               ] [master-node] stopped
[2023-01-27T05:46:44,389][INFO ][o.e.n.Node               ] [master-node] closing ...
[2023-01-27T05:46:44,401][INFO ][o.e.n.Node               ] [master-node] closed
[2023-01-27T05:46:44,402][INFO ][o.e.x.m.p.NativeController] [master-node] Native controller process has stopped - no new native processes can be started

Another error when I check the elasticsearch using systemctl status elasticsearch

× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-01-27 05:46:46 PST; 21s ago
       Docs: https://www.elastic.co
    Process: 7071 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
   Main PID: 7071 (code=exited, status=78)
        CPU: 47.165s

Jan 27 05:46:27 elk-kibana systemd[1]: Starting Elasticsearch...
Jan 27 05:46:44 elk-kibana systemd-entrypoint[7071]: bootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or di>
Jan 27 05:46:44 elk-kibana systemd-entrypoint[7071]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Jan 27 05:46:46 elk-kibana systemd-entrypoint[7071]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Jan 27 05:46:46 elk-kibana systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
Jan 27 05:46:46 elk-kibana systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jan 27 05:46:46 elk-kibana systemd[1]: Failed to start Elasticsearch.
Jan 27 05:46:46 elk-kibana systemd[1]: elasticsearch.service: Consumed 47.165s CPU time.

I tried also set to false only the xpack.security.http.ssl

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

I got this error:

[WARN ][o.e.x.s.a.ApiKeyAuthenticator] [master-node] Authentication using apikey failed - apikey authentication for id m-famYUB7t_jD9LtT5Yn encountered a failure
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:111) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:145) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:97) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:50) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:559) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:122) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:81) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:81) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:25) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:557) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:81) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:553) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:538) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$10(AuthorizationService.java:434) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:941) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:907) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:377) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:987) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:357) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:427) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:364) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:265) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:144) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:191) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:141) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:267) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:110) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:145) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:121) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.get(AbstractClient.java:480) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:216) ~[?:?]
        at org.elasticsearch.xpack.security.authc.ApiKeyService.loadApiKeyAndValidateCredentials(ApiKeyService.java:696) ~[?:?]
        at org.elasticsearch.xpack.security.authc.ApiKeyService.tryAuthenticate(ApiKeyService.java:652) ~[?:?]
        at org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator.authenticate(ApiKeyAuthenticator.java:51) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:101) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:398) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:532) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:313) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353) ~[elasticsearch-8.6.0.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-01-27T05:57:56,144][INFO ][o.e.l.LicenseService     ] [master-node] license [f698b3d2-e79b-417c-a3a3-6c5d5e789c11] mode [basic] - valid
[2023-01-27T05:57:56,145][INFO ][o.e.x.s.a.Realms         ] [master-node] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-01-27T05:57:56,165][INFO ][o.e.g.GatewayService     ] [master-node] recovered [10] indices into cluster_state
[2023-01-27T05:57:56,223][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [master-node] Node [{master-node}{TPEy3lPoSGiMVRiQtO7dHA}] is selected as the current health node.
[2023-01-27T05:57:56,833][INFO ][o.e.c.r.a.AllocationService] [master-node] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-2023.01.26-000001][0]]])." previous.health="RED" reason="shards started [[.ds-ilm-history-5-2023.01.26-000001][0]]"
[2023-01-27T05:58:00,276][WARN ][o.e.x.s.a.ApiKeyAuthenticator] [master-node] Authentication using apikey failed - unable to find apikey with id m-famYUB7t_jD9LtT5Yn
[2023-01-27T05:58:01,627][WARN ][o.e.x.s.a.ApiKeyAuthenticator] [master-node] Authentication using apikey failed - unable to find apikey with id m-famYUB7t_jD9LtT5Yn
[2023-01-27T05:58:02,862][WARN ][o.e.x.s.a.ApiKeyAuthenticator] [master-node] Authentication using apikey failed - unable to find apikey with id m-famYUB7t_jD9LtT5Yn

I also tried the same set to false the xpack.security.transport.ssl:

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

I got this error:

2023-01-27T06:01:13,608][INFO ][o.e.e.NodeEnvironment    ] [master-node] heap size [3.8gb], compressed ordinary object pointers [true]
[2023-01-27T06:01:13,670][INFO ][o.e.n.Node               ] [master-node] node name [master-node], node ID [TPEy3lPoSGiMVRiQtO7dHA], cluster name [elasticsearch], roles [remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master]
[2023-01-27T06:01:17,397][INFO ][o.e.x.s.Security         ] [master-node] Security is enabled
[2023-01-27T06:01:17,814][INFO ][o.e.x.s.a.s.FileRolesStore] [master-node] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-01-27T06:01:18,232][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [master-node] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2023-01-27T06:01:18,324][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [master-node] [controller/7455] [Main.cc@123] controller (64 bit): Version 8.6.0 (Build 85eb78d83cdd94) Copyright (c) 2022 Elasticsearch BV
[2023-01-27T06:01:18,847][INFO ][o.e.t.n.NettyAllocator   ] [master-node] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-01-27T06:01:18,872][INFO ][o.e.i.r.RecoverySettings ] [master-node] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-01-27T06:01:18,913][INFO ][o.e.d.DiscoveryModule    ] [master-node] using discovery type [multi-node] and seed hosts providers [settings]
[2023-01-27T06:01:20,211][INFO ][o.e.n.Node               ] [master-node] initialized
[2023-01-27T06:01:20,212][INFO ][o.e.n.Node               ] [master-node] starting ...
[2023-01-27T06:01:20,240][INFO ][o.e.x.s.c.f.PersistentCache] [master-node] persistent cache index loaded
[2023-01-27T06:01:20,241][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [master-node] deprecation component started
[2023-01-27T06:01:20,336][INFO ][o.e.t.TransportService   ] [master-node] publish_address {172.16.7.75:9300}, bound_addresses {[::]:9300}
[2023-01-27T06:01:20,777][INFO ][o.e.b.BootstrapChecks    ] [master-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-27T06:01:20,817][INFO ][o.e.n.Node               ] [master-node] stopping ...
[2023-01-27T06:01:20,872][INFO ][o.e.n.Node               ] [master-node] stopped
[2023-01-27T06:01:20,873][INFO ][o.e.n.Node               ] [master-node] closing ...
[2023-01-27T06:01:20,889][INFO ][o.e.n.Node               ] [master-node] closed
[2023-01-27T06:01:20,891][INFO ][o.e.x.m.p.NativeController] [master-node] Native controller process has stopped - no new native processes can be started

× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-01-27 06:01:23 PST; 38s ago
       Docs: https://www.elastic.co
    Process: 7371 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
   Main PID: 7371 (code=exited, status=78)
        CPU: 46.694s

Jan 27 06:01:03 elk-kibana systemd[1]: Starting Elasticsearch...
Jan 27 06:01:20 elk-kibana systemd-entrypoint[7371]: bootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or di>
Jan 27 06:01:20 elk-kibana systemd-entrypoint[7371]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Jan 27 06:01:23 elk-kibana systemd-entrypoint[7371]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Jan 27 06:01:23 elk-kibana systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
Jan 27 06:01:23 elk-kibana systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jan 27 06:01:23 elk-kibana systemd[1]: Failed to start Elasticsearch.
Jan 27 06:01:23 elk-kibana systemd[1]: elasticsearch.service: Consumed 46.694s CPU time.

Can someone help me where I went wrong and how to fix it?
I just started to learn Elastic Stack
Your help is very appreciated.

Cruz · January 26, 2023, 10:13pm

I followed the guide here Install Elasticsearch

Cruz · January 26, 2023, 10:16pm

This is my config for elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master-node
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
## By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: [172.16.7.75"]
#discovery.type: single-node
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 26-01-2023 04:12:23
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["elk-kibana"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Ayush_Mathur · January 27, 2023, 10:01am

Hello @Cruz , it seems you have a single node cluster and you are trying to use your own certificates for SSL.
I've encountered the logs you mentioned at the beginning and turned out it as due to system watchers trying to hit ES over HTTP: Secure connection on HTTP layer

system · February 24, 2023, 10:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.