Bulk insert HTTP 504 Gateway Timeout

Elastic Stack Elasticsearch
1

I have been experiencing a HTTP 504 Gateway Timeout from Elasticsearch during a bulk insert.
We are using Amazon Elasticsearch Service and I've read that there is a default 60 second timeout and if there are too many requests at a time, a 504 gateway timeout is likely to happen.

However, I record every ES request and the elapsed time since the request was sent and a HTTP 504 gateway timeout is only 10 ms. And the bulk insert requests during that time is only 4 requests in average.

Could you please share with me the possible reasons for this?

Here is a sample stacktrace:
_bulk: HTTP/1.1 504 GATEWAY_TIMEOUT at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354) at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.lang.Thread.run(Thread.java:745)

Thank you.

2

Elasticsearch never returns a 504 response itself, so it must be an intermediary proxy that's causing this error - likely either a proxy on your end, or if you're not using one, an issue with Amazon ESS, which I'm afraid we can't really help with. You might want to talk to Amazon's customer support.

1 Like
3

Thank you for explaining, @gbrown.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.