# There are continuous failures due to Http:500 internal server error, http:503 service unavailable and timeouts

**URL:** <https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678>\
**Category:** Elasticsearch\
**Created:** [July 10, 2020, 10:54am UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678 "2020-07-10T10:54:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sasvmware](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sasvmware](https://discuss.elastic.co/u/sasvmware)\
**Post date:** [July 10, 2020, 10:54am UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/1 "2020-07-10T10:54:32Z")

</div>

There are continuous failures while running 5x load test due to Http:500 internal server error, http:503 service unavailable and timeouts in rest api

In logs, we do not find any issue.  
3 nodes  
Memory: 16 GB  
vCpu : 4

---

<div class="post-metadata">

**Author:** ![sasvmware](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sasvmware](https://discuss.elastic.co/u/sasvmware)\
**Post date:** [July 10, 2020, 10:55am UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/2 "2020-07-10T10:55:05Z")

</div>

[2020-07-10T03:51:30,015][DEBUG][r.suppressed] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] path: /\_security/user/\_has\_privileges, params: {}  
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/\_security/user/\_has\_privileges]  
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[x-pack-core-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[x-pack-core-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:104) ~[x-pack-core-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableRestRequest.anonymousAccessDenied(AuthenticationService.java:731) ~[x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:476) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:481) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:357) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:327) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:345) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:288) [x-pack-security-7.5.1.jar:7.5.1]

---

<div class="post-metadata">

**Author:** ![sasvmware](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sasvmware](https://discuss.elastic.co/u/sasvmware)\
**Post date:** [July 10, 2020, 10:55am UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/3 "2020-07-10T10:55:27Z")

</div>

at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:328) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:269) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:252) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:390) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-7.5.1.jar:7.5.1]  
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.1.jar:7.5.1]  
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.1.jar:7.5.1]  
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.1.jar:7.5.1]  
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.1.jar:7.5.1]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]  
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]  
at java.lang.Thread.run(Thread.java:830) [?:?]  
[2020-07-10T03:51:30,165][DEBUG][o.e.x.s.a.e.ReservedRealm] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] realm [reserved] authenticated user [remote\_monitoring\_user], with roles [[remote\_monitoring\_collector, remote\_monitoring\_agent]]  
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_org\_usage\_daily\_dtls][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=2, version=502, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=189240, timestamp=1594367579993, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=189240, timestamp=1594367579993, source='peer recovery'}}}]  
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_org\_usage\_hourly\_dtls][1] no retention leases are expired from current retention leases

---

<div class="post-metadata">

**Author:** ![sasvmware](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sasvmware](https://discuss.elastic.co/u/sasvmware)\
**Post date:** [July 10, 2020, 10:55am UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/4 "2020-07-10T10:55:35Z")

</div>

[RetentionLeases{primaryTerm=2, version=1104, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=2882015, timestamp=1594367310074, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=2882015, timestamp=1594367310074, source='peer recovery'}}}]  
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_org\_usage\_daily\_dtls][2] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=1, version=500, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=188048, timestamp=1594367579993, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=188048, timestamp=1594367579993, source='peer recovery'}}}]  
[2020-07-10T03:51:30,393][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_org\_usage\_hourly\_dtls][2] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=3, version=1106, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=2884593, timestamp=1594367310074, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=2884593, timestamp=1594367310074, source='peer recovery'}}}]  
[2020-07-10T03:51:30,421][DEBUG][o.e.x.s.a.e.ReservedRealm] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] realm [reserved] authenticated user [kibana], with roles [[kibana\_system]]  
[2020-07-10T03:51:30,648][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_partner\_details\_test][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=2, version=144, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=22, timestamp=1594373130480, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=22, timestamp=1594373130480, source='peer recovery'}}}]  
[2020-07-10T03:51:30,649][DEBUG][o.e.i.s.ReplicationTracker] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] [por\_partner\_details\_test][1] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=3, version=144, leases={peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg=RetentionLease{id='peer\_recovery/MCYofunFTr-M\_H5b7Zgrpg', retainingSequenceNumber=18, timestamp=1594378230584, source='peer recovery'}, peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w=RetentionLease{id='peer\_recovery/1UhZeM3gTnujHx0YY\_mn0w', retainingSequenceNumber=18, timestamp=1594378230584, source='peer recovery'}}}]  
[2020-07-10T03:51:31,596][DEBUG][o.e.x.s.a.e.ReservedRealm] [[xxx-xxx-elkmd1.vmware.com](http://xxx-xxx-elkmd1.vmware.com)] realm [reserved] authenticated user [kibana], with roles [[kibana\_system]]

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 12, 2020, 10:37pm UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/5 "2020-07-12T22:37:29Z")

</div>

Please format your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2020, 10:37pm UTC](https://discuss.elastic.co/t/there-are-continuous-failures-due-to-http-500-internal-server-error-http-503-service-unavailable-and-timeouts/240678/6 "2020-08-09T22:37:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.