# ElasticSearch \_bulk calls resulting in "socket hang up" despite small size **URL:** \ **Category:** Elasticsearch\ **Created:** [February 27, 2013, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936 "2013-02-27T20:27:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [February 27, 2013, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936/1 "2013-02-27T20:27:37Z")
I'm using elastical to connect to ElasticSearch via node.js. In the process of profiling my app with Nodetime to attempt to improve performance, I noticed something odd. My ElasticSearch "PUT" requests to \_bulk index are frequently resulting in a "socket hang up". Furthermore, these calls are taking huge amounts of CPU time. I'm capping each \_bulk index request @ 10 items to index, and as you can see, the content-length of the requests does not even reach 50Kb, so it is hard to imagine that the size is an issue. Yet, the response time is \> 60 seconds and the CPU time is \>10+ seconds! Yikes!! \ In attempts to debug, I started running ElasticSearch in the foreground. I noticed this strange error: [2013-02-27 11:42:39,188][WARN][index.gateway.s3] [Lady Mandarin] [network][1] failed to read commit point [commit-f34] java.io.IOException: Failed to get [commit-f34] at org.elasticsearch.common.blobstore.support.AbstractBlobContainer.readBlobFully(AbstractBlobContainer.java:83) at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.buildCommitPoints(BlobStoreIndexShardGateway.java:847) at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:188) at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:160) at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:271) at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:265) at org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:1090) at org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:496) at org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:265) at org.elasticsearch.index.gateway.IndexShardGatewayService$SnapshotRunnable.run(IndexShardGatewayService.java:366) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) ``` Caused by: Status Code: 404, AWS Service: Amazon S3, AWS Request ID: ..., AWS Error Code: NoSuchKey, AWS Error Message: The specified key does not exist., S3 Extended Request ID: .... at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:548) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:288) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2632) at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:811) at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:717) at org.elasticsearch.cloud.aws.blobstore.AbstractS3BlobContainer$1.run(AbstractS3BlobContainer.java:73) ``` I'm aware that I'm using a deprecated gateway (the S3 bucket gateway). However, given that I have multiple servers running on the Amazon Cloud which need to share data (I use ElasticSearch for caching), I don't see any alternative until the ElasticSearch team releases a replacement for the S3 Bucket Gateway... Other than this problem with the \_bulk calls, I'm not seeing any problems. Searches etc. all return quickly and effectively. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 28, 2013, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936/2 "2013-02-28T16:19:18Z")
Which error are you seeing? You don't need to use s3 gateway to have a cluster of nodes setup on AWS…, s3 gateway simply consistently copies the data to s3, while the local gateway will know how to restore its state from each instance local storage (EBS or ephemeral). On Feb 27, 2013, at 9:27 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > I'm using elastical to connect to Elasticsearch via node.js. > > In the process of profiling my app with Nodetime to attempt to improve > performance, I noticed something odd. My Elasticsearch "PUT" requests to > \_bulk index are frequently resulting in a "socket hang up". Furthermore, > these calls are taking huge amounts of CPU time. > > I'm capping each \_bulk index request @ 10 items to index, and as you can > see, the content-length of the requests does not even reach 50Kb, so it is > hard to imagine that the size is an issue. Yet, the response time is \> 60 > seconds and the CPU time is \>10+ seconds! Yikes!! > > [http://elasticsearch-users.115913.n3.nabble.com/file/n4030677/EoIWe.png](http://elasticsearch-users.115913.n3.nabble.com/file/n4030677/EoIWe.png) > > In attempts to debug, I started running Elasticsearch in the foreground. I > noticed this strange error: > > I'm aware that I'm using a deprecated gateway (the S3 bucket gateway). > However, given that I have multiple servers running on the Amazon Cloud > which need to share data (I use Elasticsearch for caching), I don't see any > alternative until the Elasticsearch team releases a replacement for the S3 > Bucket Gateway... > > Other than this problem with the \_bulk calls, I'm not seeing any problems. > Searches etc. all return quickly and effectively. > > -- > View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-bulk-calls-resulting-in-socket-hang-up-despite-small-size-tp4030677.html](http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-bulk-calls-resulting-in-socket-hang-up-despite-small-size-tp4030677.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 28, 2013, 4:25pm UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936/3 "2013-02-28T16:25:57Z")
It looks like you run into internal shard bulk response timeouts which are by default 60 seconds. There should be something in the logs. Check if you can successfully index documents to all the cluster nodes. Best regards, Jörg Am 27.02.13 21:27, schrieb inZania: > Yet, the response time is \> 60 > seconds and the CPU time is \>10+ seconds! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 4:10pm UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936/4 "2013-03-05T16:10:48Z")
kimchy, Data persistency between the instances is of very high importance to me. Eg, I might have one server which indexes a new document, and 30s later I'll need to search for that document from the other server... and then maybe one of the servers might go down (it is an auto-scaling array on the cloud, so servers go up and down all the time), and I still need the data to be available. I thought this wasn't possible without S3 as a shared data store -- am I wrong? If so, where can I find a tutorial on how to do this? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:48am UTC](https://discuss.elastic.co/t/elasticsearch--bulk-calls-resulting-in-socket-hang-up-despite-small-size/10936/5 "2017-07-06T02:48:17Z")