# Error while inserting bulk data in big size via logstash to elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [October 24, 2017, 10:25am UTC](https://discuss.elastic.co/t/error-while-inserting-bulk-data-in-big-size-via-logstash-to-elasticsearch/105031 "2017-10-24T10:25:02Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![PinkiMondal](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@PinkiMondal](https://discuss.elastic.co/u/PinkiMondal)\ **Post date:** [October 24, 2017, 10:25am UTC](https://discuss.elastic.co/t/error-while-inserting-bulk-data-in-big-size-via-logstash-to-elasticsearch/105031/1 "2017-10-24T10:25:02Z")
Hi All I am using elasticsearch version 5.4.1 and I have 8 Million records combined all the indices. Now I have one index which contains **21 lakhs** records and I want to insert more records in that Index First bulk request containing **1 Lakh** records I am getting: **[2017-10-24T12:29:47,946][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 (** **{"type"=\>"unavailable\_shards\_exception", "reason"=\>"[jdbc\_metadata\_pt\_new\_logs][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[jdbc\_metadata\_pt\_new\_logs][2]] containing [27] requests]"}** **)** **[2017-10-24T12:29:47,946][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 (** **{"type"=\>"unavailable\_shards\_exception", "reason"=\>"[jdbc\_metadata\_pt\_new\_logs][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[jdbc\_metadata\_pt\_new\_logs][2]] containing [27] requests]"}** **)** ... **[2017-10-24T12:29:47,946][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 (** **{"type"=\>"unavailable\_shards\_exception", "reason"=\>"[jdbc\_metadata\_pt\_new\_logs][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[jdbc\_metadata\_pt\_new\_logs][2]] containing [27] requests]"}** **)** **[2017-10-24T12:29:47,946][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 (** **{"type"=\>"unavailable\_shards\_exception", "reason"=\>"[jdbc\_metadata\_pt\_new\_logs][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[jdbc\_metadata\_pt\_new\_logs][2]] containing [27] requests]"}** **)** **[2017-10-24T12:29:47,946][ERROR][logstash.outputs.elasticsearch] Retrying individual actions** **[2017-10-24T13:08:30,172][ERROR][logstash.outputs.elasticsearch] Action** ... **[2017-10-24T13:08:30,173][ERROR][logstash.outputs.elasticsearch] Action** **[2017-10-24T13:09:29,929][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://url/](http://url/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://url/](http://url/), :error\_message=\>"Elasticsearch Unreachable: [[http://url/](http://url/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}** **[2017-10-24T13:09:29,930][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://url/](http://url/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2}** **[2017-10-24T13:09:29,930][DEBUG][logstash.outputs.elasticsearch] Failed actions for last bad bulk request! {:actions=\>[["index", {:\_id=\>"111847", :\_index=\>"jdbc\_metadata\_pt\_new\_logs", :\_type=\>"jdbc\_metadata\_pt\_new\_logs", :\_routing=\>nil}, 2017-10-24T07:27:10.090Z %{host} %{message}] ... ["index", {:\_id=\>"1570686", :\_index=\>"jdbc\_metadata\_pt\_new\_logs", :\_type=\>"jdbc\_metadata\_pt\_new\_logs", :\_routing=\>nil}, 2017-10-24T07:27:10.222Z %{host} %{message}]]}** **[2017-10-24T13:09:29,981][WARN][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=\>#\}** **[2017-10-24T13:09:29,981][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error\_message=\>"No Available connections", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will\_retry\_in\_seconds=\>2}** **[2017-10-24T13:09:30,439][WARN][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=\>#\}** **[2017-10-24T13:09:30,440][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error\_message=\>"No Available connections", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will\_retry\_in\_seconds=\>2}** [2017-10-24T13:09:30,460][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck ... Caused by: java.io.IOException: No space left on device\*\* at java.io.FileOutputStream.writeBytes(Native Method)\*\* at java.io.FileOutputStream.write(FileOutputStream.java:326)\*\* at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:211) **Then I tried bulk request with 100 records Then I am below exception:** [2017-10-24T12:37:43,498][WARN][logstash.shutdownwatcher] {"inflight\_count"=\>3, "stalling\_thread\_info"=\>{"other"=\>[{"thread\_id"=\>24, "name"=\>"[main]\>worker3", "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:50:in `call'"}]}} [2017-10-24T12:37:48,497][WARN][logstash.shutdownwatcher] {"inflight_count"=>3, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:50:in`call'"}]}} ... [2017-10-24T12:38:38,498][WARN][logstash.shutdownwatcher] {"inflight\_count"=\>3, "stalling\_thread\_info"=\>{"other"=\>[{"thread\_id"=\>24, "name"=\>"[main]\>worker3", "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:50:in `call'"}]}} [2017-10-24T12:38:43,240][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=\>"unavailable\_shards\_exception", "reason"=\>"[jdbc\_metadata\_pt\_new\_logs][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[jdbc\_metadata\_pt\_new\_logs][2]] containing [index {[jdbc\_metadata\_pt\_new\_logs][jdbc\_metadata\_pt\_new\_logs][48727], source[n/a, actual length: [4.8kb], max length: 2kb]}]]"}) [2017-10-24T12:38:43,241][ERROR][logstash.outputs.elasticsearch] Retrying individual actions [2017-10-24T12:38:43,241][ERROR][logstash.outputs.elasticsearch] Action [2017-10-24T12:38:48,497][WARN][logstash.shutdownwatcher] {"inflight\_count"=\>3, "stalling\_thread\_info"=\>{"other"=\>[{"thread\_id"=\>24, "name"=\>"[main]\>worker3", "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:84:in `sleep'"}]}} Please tell why I am getting this error ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 24, 2017, 1:43pm UTC](https://discuss.elastic.co/t/error-while-inserting-bulk-data-in-big-size-via-logstash-to-elasticsearch/105031/2 "2017-10-24T13:43:23Z")
> Caused by: java.io.IOException: No space left on device You're out of disk space. I'm not sure if this is in Logstash or ES, but either way the ES cluster's health is bad. Get ES in shape first. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 21, 2017, 1:43pm UTC](https://discuss.elastic.co/t/error-while-inserting-bulk-data-in-big-size-via-logstash-to-elasticsearch/105031/3 "2017-11-21T13:43:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.