# Replication timeouts
**URL:** https://discuss.elastic.co/t/replication-timeouts/17231
**Category:** Elasticsearch
**Created:** [April 28, 2014, 10:22am UTC](https://discuss.elastic.co/t/replication-timeouts/17231 "2014-04-28T10:22:10Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Michael\_Salmon](https://discuss.elastic.co/u/Michael_Salmon)
#### Post date: [April 28, 2014, 10:22am UTC](https://discuss.elastic.co/t/replication-timeouts/17231/1 "2014-04-28T10:22:10Z")
I have started getting some timeouts during replication and I am unsure of
how to proceed. The index is about 500 million documents or 45GB spread
over 8 shards and created by a jdbc river. The timeout is occurring
during index/shard/recovery/prepareTranslog. It seems that the limit of 15
minutes is hard coded or I would have tried changing that.
There are several parameters relate to index recovery but I'm not sure how
they affect performance. Has anyone any suggestions?
/Michael
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [April 28, 2014, 10:48am UTC](https://discuss.elastic.co/t/replication-timeouts/17231/2 "2014-04-28T10:48:22Z")
Is it possible to post the full timeout exception?
Do you run JDBC river with replica level 0 and add replica later after
river completion?
I saw this in the past and I'm not sure if this is related to tight
resources.
In the next JDBC river version there will be more convenient control of
bulk index settings (automatic replica level 0, refresh disabling,
re-enabling of refresh & replica afterwards).
Jörg
On Mon, Apr 28, 2014 at 12:22 PM, Michael Salmon
[michael.salmon@inovia.nu](mailto:michael.salmon@inovia.nu)wrote:
> I have started getting some timeouts during replication and I am unsure of
> how to proceed. The index is about 500 million documents or 45GB spread
> over 8 shards and created by a jdbc river. The timeout is occurring
> during index/shard/recovery/prepareTranslog. It seems that the limit of 15
> minutes is hard coded or I would have tried changing that.
>
> There are several parameters relate to index recovery but I'm not sure how
> they affect performance. Has anyone any suggestions?
>
> /Michael
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHSY09g3Dq2tgs18KPiOy82BVtNhoyWVKw-f4OmRjjn-A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHSY09g3Dq2tgs18KPiOy82BVtNhoyWVKw-f4OmRjjn-A%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Michael\_Salmon](https://discuss.elastic.co/u/Michael_Salmon)
#### Post date: [April 28, 2014, 12:14pm UTC](https://discuss.elastic.co/t/replication-timeouts/17231/3 "2014-04-28T12:14:26Z")
[2014-04-28 13:40:15,039][WARN][cluster.action.shard] [eis05]
[ds\_clearcase-vob-heat-analyzer][2] sending failed shard for
[ds\_clearcase-vob-heat-analyzer][2], node[QyeTlW2YQbG27zrsdjBBGA], [R],
s[INITIALIZING], indexUUID [ms7jQeuMQduNIHCmjxsKjQ], reason [Failed to
start shard, message
[RecoveryFailedException[[ds\_clearcase-vob-heat-analyzer][2]: Recovery
failed from
[eis09][p8-\_fzHeTR22pSlsBsYm8A][eis09.rnditlab.ericsson.se][inet[/137.58.184.239:9300]]{datacenter=PoCC}
into
[eis05][QyeTlW2YQbG27zrsdjBBGA][eis05.rnditlab.ericsson.se][inet[eis05.rnditlab.ericsson.se/137.58.184.235:9300]]{datacenter=PoCC}];
nested:
RemoteTransportException[[eis09][inet[/137.58.184.239:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[ds\_clearcase-vob-heat-analyzer][2]
Phase[2] Execution failed]; nested:
ReceiveTimeoutTransportException[[eis05][inet[/137.58.184.235:9300]][index/shard/recovery/prepareTranslog]
request\_id [6809886] timed out after [900000ms]]; ]]
[2014-04-28 14:00:11,614][WARN][indices.cluster] [eis05]
[ds\_clearcase-vob-heat-analyzer][0] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[ds\_clearcase-vob-heat-analyzer][0]: Recovery failed from
[eis07][Q8ZWgDIXRGiUej1oMoH8Jg][eis07.rnditlab.ericsson.se][inet[/137.58.184.237:9300]]{datacenter=PoCC}
into
[eis05][QyeTlW2YQbG27zrsdjBBGA][eis05.rnditlab.ericsson.se][inet[eis05.rnditlab.ericsson.se/137.58.184.235:9300]]{datacenter=PoCC}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.RemoteTransportException:
[eis07][inet[/137.58.184.237:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[ds\_clearcase-vob-heat-analyzer][0] Phase[2] Execution failed
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1098)
at
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:627)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:117)
at
org.elasticsearch.indices.recovery.RecoverySource.access$1600(RecoverySource.java:61)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:337)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:323)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException:
[eis05][inet[/137.58.184.235:9300]][index/shard/recovery/prepareTranslog]
request\_id [154592652] timed out after [900000ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
... 3 more
The river has been running for some time copying new documents from the db
into es. The problem as I see it is that it is too big to copy in 15
minutes.
/Michael
On Monday, 28 April 2014 12:48:22 UTC+2, Jörg Prante wrote:
> Is it possible to post the full timeout exception?
>
> Do you run JDBC river with replica level 0 and add replica later after
> river completion?
>
> I saw this in the past and I'm not sure if this is related to tight
> resources.
>
> In the next JDBC river version there will be more convenient control of
> bulk index settings (automatic replica level 0, refresh disabling,
> re-enabling of refresh & replica afterwards).
>
> Jörg
>
> On Mon, Apr 28, 2014 at 12:22 PM, Michael Salmon \
>
> > wrote:
>
> > I have started getting some timeouts during replication and I am unsure
> > of how to proceed. The index is about 500 million documents or 45GB spread
> > over 8 shards and created by a jdbc river. The timeout is occurring
> > during index/shard/recovery/prepareTranslog. It seems that the limit of 15
> > minutes is hard coded or I would have tried changing that.
> >
> > There are several parameters relate to index recovery but I'm not sure
> > how they affect performance. Has anyone any suggestions?
> >
> > /Michael
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c15c79b-b9c5-4f7f-98f1-68b692f86fc6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c41772d0-7251-4f74-81b1-7f1058ed24f6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c41772d0-7251-4f74-81b1-7f1058ed24f6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:33am UTC](https://discuss.elastic.co/t/replication-timeouts/17231/4 "2017-07-06T01:33:08Z")