iam seeing this error in the logs.It was working fine till 1 day back.Whta could be the issue here.
i restarted the kibana.Still the same issue.ECerything was properly setup.
{"type":"log","@timestamp":"2020-12-08T06:53:00Z","tags":["reporting","warning"],"pid":1,"message":"Reporting plugin self-check failed. Please check the Kibana Reporting settings. [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_cluster/settings","query":{"include_defaults":true},"statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [elastic]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [elastic]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}"}
{"type":"log","@timestamp":"2020-12-08T06:53:00Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:00Z","tags":["warning","maps"],"pid":1,"message":"Error scheduling telemetry task, received NotInitialized: Tasks cannot be scheduled until after task manager is initialized!"}
{"type":"log","@timestamp":"2020-12-08T06:53:00Z","tags":["warning","telemetry"],"pid":1,"message":"Error scheduling task, received NotInitialized: Tasks cannot be scheduled until after task manager is initialized!"}
{"type":"log","@timestamp":"2020-12-08T06:53:03Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:06Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:09Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:12Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:15Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:18Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:21Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:24Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:27Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:28Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [elastic]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [elastic]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}"}
{"type":"log","@timestamp":"2020-12-08T06:53:30Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:33Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:36Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
{"type":"log","@timestamp":"2020-12-08T06:53:39Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}
Has something changed with the default user? Can you query the Elasticsearch APIs using the details you have set?
This is what i could see in my elastic client node.Nothing has been changed.
{"type": "server", "timestamp": "2020-12-08T07:58:35,729+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "Gs7d-Bl0Qsi1jjqz0uv9HA", "node.id": "KWjHxe7_RH-6VBs5tGSm_w", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-08T07:58:37,335+0000", "level": "ERROR", "component": "o.e.x.s.a.e.ReservedRealm", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "Gs7d-Bl0Qsi1jjqz0uv9HA", "node.id": "KWjHxe7_RH-6VBs5tGSm_w", "message": "failed to retrieve password hash for reserved user [elastic]" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable",
"at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:181) ~[x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:525) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:212) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:93) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:364) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:407) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:335) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:345) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:288) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:360) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:269) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:252) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:390) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-7.3.0.jar:7.3.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:344) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) [elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-7.3.0.jar:7.3.0]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.3.0.jar:7.3.0]",
"at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.3.0.jar:7.3.0]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
{"type": "server", "timestamp": "2020-12-08T07:58:37,336+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "Gs7d-Bl0Qsi1jjqz0uv9HA", "node.id": "KWjHxe7_RH-6VBs5tGSm_w", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
You still have auth issues, so you need to check you can connect to Elasticsearch using the API and the details you have.
i tried with curl..i keep getting this
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
Looks strange to me.Nothing is chnaged suddenly elastic user is failing.
Sounds like something has changed then. Do you have other users you can try?
i tried with another usr also with no luck
curl -u sudhakar 'http://elasticsearch-client.logging.svc.cluster.local:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'sudhakar':
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "unable to authenticate user [sudhakar] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "unable to authenticate user [sudhakar] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
THis is Elasticsearch 7.3 and iam running a docker image.
Ok this is a bit of a problem.
Did something happen to your cluster recently?
elasticsearch-client-56db759bc-ds4c8 1/1 Running 0 15d
elasticsearch-data-0 1/1 Running 0 3d19h
elasticsearch-master-645ccc5b4c-nl2fp 1/1 Running 0 15d
not that iam aware of
iam seeing this in my master node
{"type": "server", "timestamp": "2020-12-08T22:37:31,434+0000", "level": "ERROR", "component": "o.e.x.m.c.c.ClusterStatsCollector", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master", "cluster.uuid": "Gs7d-Bl0Qsi1jjqz0uv9HA", "node.id": "Yn6TOUnHT6WlKbBW8qhnFw", "message": "collector [cluster_stats] failed to collect data" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable",
"at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:181) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getUserCount(NativeUsersStore.java:170) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.esnative.NativeRealm.lambda$usageStats$1(NativeRealm.java:53) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$usageStats$5(CachingUsernamePasswordRealm.java:203) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.xpack.core.security.authc.Realm.usageStats(Realm.java:137) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.usageStats(CachingUsernamePasswordRealm.java:201) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.esnative.NativeRealm.usageStats(NativeRealm.java:52) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.Realms.usageStats(Realms.java:268) ~[?:?]",
"at org.elasticsearch.xpack.security.SecurityFeatureSet.usage(SecurityFeatureSet.java:143) ~[?:?]"
I am not sure how to recover from this sorry. Hopefully one of the security engineers can take a look.
If i restart my nodes..will i loose data?
At this point there's a chance you've already lost data.
The only thing we know is that one of your indices (.security-7) is missing its primary shard. Perhaps other indices are missing as well, we won't know until you can get the cluster into a state where you can access the management APIs and see what's going on.
If you have a paid support contract, you should log a case, so that you can get expert help working out what's happened and how to resolve it.
If not...
- You need to work out whether you have the file realm enabled. By default, you will, but if you have made changes to your
elasticsearch.ymland have changed the realm configuration, then you might not.- If it is not enabled then you will need to enable it on 1 node and restart that node.
- Use the
elasticsearch-usersutility to add a new superuser to your clusterelasticsearch-users useradd admin -p password -r superuser - Use that new
adminuser to check the cluster health and the status of your shards.
i have restarted my nodes
my elastic master has this status
{"type": "server", "timestamp": "2020-12-08T22:56:38,781+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "dg6m_iePT1CW2ImOFE6drA", "message": "Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-7-2020.12.08][0]] ...])." }
My elastic lcient has this error
{"type": "server", "timestamp": "2020-12-09T03:36:39,597+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-09T03:36:40,897+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-09T03:36:42,178+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-09T03:36:42,178+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-09T03:36:42,269+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
{"type": "server", "timestamp": "2020-12-09T03:36:42,269+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
i have the basic licesnse.i have setup in other test environments and everything is working fine.Only this environment its having this strange issue.
I have restarted kibana
{"type":"response","@timestamp":"2020-12-09T02:37:29Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"example.com","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"10.240.0.8","userAgent":"10.240.0.8"},"res":{"statusCode":404,"responseTime":19,"contentLength":9},"message":"GET / 404 19ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:29Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/TP/public/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.6","userAgent":"10.240.0.6"},"res":{"statusCode":404,"responseTime":100,"contentLength":9},"message":"GET /TP/public/index.php 404 100ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:30Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/TP/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.5","userAgent":"10.240.0.5"},"res":{"statusCode":404,"responseTime":19,"contentLength":9},"message":"GET /TP/index.php 404 19ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:30Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/thinkphp/html/public/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.8","userAgent":"10.240.0.8"},"res":{"statusCode":404,"responseTime":19,"contentLength":9},"message":"GET /thinkphp/html/public/index.php 404 19ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:31Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/html/public/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.8","userAgent":"10.240.0.8"},"res":{"statusCode":404,"responseTime":17,"contentLength":9},"message":"GET /html/public/index.php 404 17ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:31Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/public/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.6","userAgent":"10.240.0.6"},"res":{"statusCode":404,"responseTime":99,"contentLength":9},"message":"GET /public/index.php 404 99ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:32Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/TP/html/public/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.5","userAgent":"10.240.0.5"},"res":{"statusCode":404,"responseTime":18,"contentLength":9},"message":"GET /TP/html/public/index.php 404 18ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:33Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/elrekt.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.6","userAgent":"10.240.0.6"},"res":{"statusCode":404,"responseTime":18,"contentLength":9},"message":"GET /elrekt.php 404 18ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:33Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/index.php","method":"get","headers":{"host":"20.186.48.230","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.240.0.5","userAgent":"10.240.0.5"},"res":{"statusCode":404,"responseTime":100,"contentLength":9},"message":"GET /index.php 404 100ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T02:44:34Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"20.186.48.230:80","user-agent":"Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)","connection":"close","accept-encoding":"gzip"},"remoteAddress":"10.244.5.1","userAgent":"10.244.5.1"},"res":{"statusCode":404,"responseTime":20,"contentLength":9},"message":"GET / 404 20ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T03:13:13Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"20.186.48.230:80","user-agent":"Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)"},"remoteAddress":"10.240.0.6","userAgent":"10.240.0.6"},"res":{"statusCode":404,"responseTime":102,"contentLength":9},"message":"GET / 404 102ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-09T03:22:53Z","tags":,"pid":1,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"20.186.48.230:80","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-length":"0"},"remoteAddress":"10.240.0.5","userAgent":"10.240.0.5"},"res":{"statusCode":404,"responseTime":20,"contentLength":9},"message":"GET / 404 20ms - 9.0B"}
iam kibana UI is not laoding properly
i have reset new password also for elastic during restarting of my elasric cluster
Did you test those new details to make sure they work?
Did you update Kibana?
Yes updated kibana also.
My kibana UI is not loading
Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'nonce-rKogLRTK6yL0L/+A'". Either the 'unsafe-inline' keyword, a hash ('sha256-SHHSeLc0bp6xt4BoVVyUy+3IbVqp3ujLaR+s+kSP5UI='), or a nonce ('nonce-...') is required to enable inline execution.
bootstrap.js:10 ^ A single error about an inline script not firing due to content security policy is expected!
index.light.css:1 Failed to load resource: the server responded with a status of 503 ()
index.light.css:1 Failed to load resource: the server responded with a status of 503 ()
bootstrap.js:32 Uncaught TypeError: Cannot read property 'dataset' of null
at HTMLLinkElement.failure (bootstrap.js:32)
index.light.css:1 Failed to load resource: the server responded with a status of 503 ()
index.light.css:1 Failed to load resourc
and elastic lient has the error pasted in below.
{"type": "server", "timestamp": "2020-12-09T03:36:42,178+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-client", "cluster.uuid": "VPIMafK0Sm2-WBwsS4u_iA", "node.id": "r-ZQVtM9Q0OATheKUumOag", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }
My cluster status is green. this is the o/p of the cluster health
{"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":1,"active_primary_shards":7,"active_shards":7,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.