There are continuous failures while running 5x load test due to Http:500 internal server error, http:503 service unavailable and timeouts in rest api
In logs, we do not find any issue.
3 nodes
Memory: 16 GB
vCpu : 4
[2020-07-10T03:51:30,015][DEBUG][r.suppressed ] [xxx-xxx-elkmd1.vmware.com] path: /_security/user/_has_privileges, params: {}
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/_security/user/_has_privileges]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[x-pack-core-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[x-pack-core-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:104) ~[x-pack-core-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableRestRequest.anonymousAccessDenied(AuthenticationService.java:731) ~[x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:476) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:481) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:357) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:327) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:345) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:288) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:328) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:269) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:252) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:390) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.1.jar:7.5.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.1.jar:7.5.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.1.jar:7.5.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-07-10T03:51:30,165][DEBUG][o.e.x.s.a.e.ReservedRealm] [xxx-xxx-elkmd1.vmware.com] realm [reserved] authenticated user [remote_monitoring_user], with roles [[remote_monitoring_collector, remote_monitoring_agent]]
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_org_usage_daily_dtls][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=2, version=502, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=189240, timestamp=1594367579993, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=189240, timestamp=1594367579993, source='peer recovery'}}}]
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_org_usage_hourly_dtls][1] no retention leases are expired from current retention leases
[RetentionLeases{primaryTerm=2, version=1104, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=2882015, timestamp=1594367310074, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=2882015, timestamp=1594367310074, source='peer recovery'}}}]
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_org_usage_daily_dtls][2] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=1, version=500, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=188048, timestamp=1594367579993, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=188048, timestamp=1594367579993, source='peer recovery'}}}]
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_org_usage_hourly_dtls][2] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=3, version=1106, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=2884593, timestamp=1594367310074, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=2884593, timestamp=1594367310074, source='peer recovery'}}}]
[2020-07-10T03:51:30,421][DEBUG][o.e.x.s.a.e.ReservedRealm] [xxx-xxx-elkmd1.vmware.com] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2020-07-10T03:51:30,648][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_partner_details_test][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=2, version=144, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=22, timestamp=1594373130480, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=22, timestamp=1594373130480, source='peer recovery'}}}]
[2020-07-10T03:51:30,649][DEBUG][o.e.i.s.ReplicationTracker] [xxx-xxx-elkmd1.vmware.com] [por_partner_details_test][1] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=3, version=144, leases={peer_recovery/MCYofunFTr-M_H5b7Zgrpg=RetentionLease{id='peer_recovery/MCYofunFTr-M_H5b7Zgrpg', retainingSequenceNumber=18, timestamp=1594378230584, source='peer recovery'}, peer_recovery/1UhZeM3gTnujHx0YY_mn0w=RetentionLease{id='peer_recovery/1UhZeM3gTnujHx0YY_mn0w', retainingSequenceNumber=18, timestamp=1594378230584, source='peer recovery'}}}]
[2020-07-10T03:51:31,596][DEBUG][o.e.x.s.a.e.ReservedRealm] [xxx-xxx-elkmd1.vmware.com] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.