# Failed Action **URL:** \ **Category:** Elasticsearch\ **Created:** [February 8, 2017, 9:33pm UTC](https://discuss.elastic.co/t/failed-action/74433 "2017-02-08T21:33:49Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 8, 2017, 9:33pm UTC](https://discuss.elastic.co/t/failed-action/74433/1 "2017-02-08T21:33:49Z")
Last night I upgraded my Elasticsearch data nodes from 4 CPUs to 8 CPUs and now I am getting these errors from my Logstash shippers. Please help!!!!! ``` [2017-02-08T15:31:14,344][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>500, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.02.08", :_type=>"logs", :_routing=>nil}, ########, :response=>{"index"=>{"_index"=>"logstash-2017.02.08", "_type"=>"logs", "_id"=>"AVofozXYJ8M9dMOnBjdZ", "status"=>500, "error"=>{"type"=>"illegal_state_exception", "reason"=>"Message not fully read (request) for requestId [151918], action [indices:data/write/bulk[s]], available [30016]; resetting"}}}} ``` ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 8, 2017, 10:10pm UTC](https://discuss.elastic.co/t/failed-action/74433/2 "2017-02-08T22:10:47Z")
Sooooooo, I deleted my indexes, then I reinstalled Elasticsearch. STILL getting this error. Please help. ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 9, 2017, 4:32am UTC](https://discuss.elastic.co/t/failed-action/74433/3 "2017-02-09T04:32:18Z")
So, I gave up. I formatted and started over. I rebuilt my entire cluster (6 data nodes, 3 master nodes, and 1 Kibana node). I am STILL getting these errors. This has gotten too ridiculous for words. I upgraded my data nodes last night from 4 CPUs to 8 CPUs and everything has gone downhill from there. ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 19, 2017, 3:25am UTC](https://discuss.elastic.co/t/failed-action/74433/4 "2017-02-19T03:25:28Z")
I have noticed I am still getting these errors. Do anyone have any ideas? ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 22, 2017, 10:59pm UTC](https://discuss.elastic.co/t/failed-action/74433/5 "2017-02-22T22:59:27Z")
Anyone? Bueller? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 27, 2017, 9:23am UTC](https://discuss.elastic.co/t/failed-action/74433/6 "2017-02-27T09:23:55Z")
Hey, instead of just bumping this, you could actively provide some more information. I am sure, that increasing the number of CPUs is not the core of your problem. - Which logstash version are you using? - Which JVM do you run on? - How did you install logstash? (RPM, DEB, zip, etc) - Which Elasticsearch version are you using? - Which JVM do you run on? - How did you install Elasticsearch? (RPM, DEB, zip, etc) - Did you check the Elasticsearch log files? - How does you logstash config look like? - How does your Elasticsearch config look like? What is the difference from a standard configuration? - Can you reproduce this with a regular HTTP request? --Alex ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [February 27, 2017, 4:40pm UTC](https://discuss.elastic.co/t/failed-action/74433/7 "2017-02-27T16:40:37Z")
> [@spinscale](#): > > instead of just bumping this, you could actively provide some more information. I have had several posts related to errors like this issue up. So, sorry if I did not include everything in this one. All Logstash versions are 5.2.1 and the ES cluster servers are 5.2.1 as well. ``` master name version role disk.avail heap.max ram.max ram.current cpu uptime jdk - DATANODE-01 5.2.1 d 1.3tb 29.9gb 39.2gb 39gb 25 3.9d 1.8.0_121 - DATANODE-02 5.2.1 d 1.3tb 29.9gb 39.2gb 38gb 14 3.9d 1.8.0_121 - DATANODE-03 5.2.1 d 1.4tb 29.9gb 39.2gb 38.8gb 32 3.9d 1.8.0_121 - DATANODE-04 5.2.1 d 1.3tb 29.9gb 39.2gb 38.8gb 28 3.9d 1.8.0_121 - DATANODE-06 5.2.1 d 1.4tb 29.9gb 39.2gb 38.2gb 1 3.9d 1.8.0_121 - KBNANODE-01 5.2.1 - 39.1gb 7.9gb 15.6gb 10gb 0 3.9d 1.8.0_121 - MSTRNODE-01 5.2.1 m 39.9gb 7.9gb 15.6gb 9.2gb 0 3.9d 1.8.0_121 - MSTRNODE-02 5.2.1 m 39.9gb 7.9gb 15.6gb 9.2gb 0 3.9d 1.8.0_121 * MSTRNODE-03 5.2.1 m 39.8gb 7.9gb 15.6gb 9.2gb 0 3.9d 1.8.0_121 ``` I installed Logstash on the shippers via apt-get and ES on the clusters via apt-get. Not sure which file you are referring to for the Logstash config. (Logstash.yml?) Same for Elasticsearch (Elasticsearch.yml?) I do not know how to reproduce the error. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 27, 2017, 6:07pm UTC](https://discuss.elastic.co/t/failed-action/74433/8 "2017-02-27T18:07:19Z")
so, have you checked all the log files of your nodes for the above error messages? It has to come from somewhere and it is likely to be logged. Is there a stack trace? Thats one of the more important bits to get. Is there any special configuration in the elasticsearch yml file. Is there any special configuration in the logstash configuration file? Can you show the elasticsearch output configuration section? ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [March 3, 2017, 5:05pm UTC](https://discuss.elastic.co/t/failed-action/74433/9 "2017-03-03T17:05:10Z")
> [@spinscale](#): > > so, have you checked all the log files of your nodes for the above error messages? It has to come from somewhere and it is likely to be logged. Is there a stack trace? Thats one of the more important bits to get. I have not seen any from any of the ES cluster servers. > [@spinscale](#): > > Is there any special configuration in the elasticsearch yml file. Nothing unique. Basic options configured. > [@spinscale](#): > > Is there any special configuration in the logstash configuration file? Can you show the elasticsearch output configuration section? Sure, here is my Logstash output: ``` ########## # ELASTICSAERCH Output Parameters ########## output { elasticsearch { hosts => ["http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200"] index => "logstash-%{+YYYY.MM.dd}" } } ################################################## ``` This is the same output configuration for all 30 Logstash shippers. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [March 5, 2017, 9:12pm UTC](https://discuss.elastic.co/t/failed-action/74433/10 "2017-03-05T21:12:28Z")
Hey, given the error message, I am still pretty sure somewhere must be an exception in the logs - at least I hope so to move forward. Either in the node which received the bulk requests or one of the data nodes where parts of the bulk request were forwarded to. Can you search for `Message not fully read` on all of your nodes in the logs? I am pretty surprised by that message, given you are using the same version on all of your nodes, so I would like to get my hands on a stack trace. The message indicates, that on the transport protocol level an unexpected message length was sent from one node to another. On HTTP level from logstash to ES this means everything was fine. --Alex ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [March 6, 2017, 3:30pm UTC](https://discuss.elastic.co/t/failed-action/74433/11 "2017-03-06T15:30:44Z")
I will do some digging to see what I can find. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 3, 2017, 3:30pm UTC](https://discuss.elastic.co/t/failed-action/74433/12 "2017-04-03T15:30:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.