# ELK Server Failed **URL:** \ **Category:** Logstash\ **Created:** [August 6, 2018, 2:07pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179 "2018-08-06T14:07:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 2:07pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/1 "2018-08-06T14:07:07Z")
Hi, My ELK server failed 2 days again with an issue in Elasticsearch for 'Too Many Files Open' I managed to resolved this moving out old indexes. Now I can access Kibana, but im not seeing any new indexes being created nor any data being received. I have received the pipeline blocked error in my logstash log files, so following information I increased my congestion\_threashold to 400 (large number to stop the circuit breaker) Im not seeing anything being indexed and receive message=\>"retrying failed action with response code: 503", :level=\>:warn} in the logs Any suggestions? Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 6, 2018, 2:10pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/2 "2018-08-06T14:10:01Z")
What is the error in the elasticsearch logs? ---
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 2:20pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/3 "2018-08-06T14:20:23Z")
None - Last entry is I just restarted the server a minute or 2 before: [2018-08-06 15:14:31,619][INFO][node] [nms1] started [2018-08-06 15:14:47,665][INFO][gateway] [nms1] recovered [413] indices into cluster\_state ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2018, 2:24pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/4 "2018-08-06T14:24:11Z")
How much heap do you have assigned? How many indices and shards are there in the cluster? ---
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 2:29pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/5 "2018-08-06T14:29:52Z")
cluster\_name":"nms", "status":"red", "timed\_out":false, "number\_of\_nodes":1, "number\_of\_data\_nodes":1, "active\_primary\_shards":2043, "active\_shards":2043, "relocating\_shards":0, "initializing\_shards":0, "unassigned\_shards":2077, "delayed\_unassigned\_shards":0, "number\_of\_pending\_tasks":0, "number\_of\_in\_flight\_fetch":0, "task\_max\_waiting\_in\_queue\_millis":0, "active\_shards\_percent\_as\_number":49.5873786407767} heap size [19.9gb] Some detail above... So... I have noticed my cluster is red and it's loaded only 49% of my shards... Could this be due to moving and not using XDELETE on my old indexs? I have put 1 back and restarted and can see the active primary shards has increased Would this stop the data getting in? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2018, 3:02pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/6 "2018-08-06T15:02:42Z")
I think you have far too many shards for a cluster that size. Please read [this blog post around shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). ---
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 3:12pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/7 "2018-08-06T15:12:28Z")
I plan to do this after I can get the server working again. I have had this working with more shards so I should be able to get it to a workable state first. Where else can I go with this? ---
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 3:58pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/8 "2018-08-06T15:58:25Z")
Ok so I have got a little further... I shutdown ELK, moved out some indexes from around the time the system crashed and restarted. I got a burst of data into the system. The my OSSEC servers received a connection refused error. ---
**Author:** ![Chris\_Billett](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Post date:** [August 6, 2018, 4:53pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/9 "2018-08-06T16:53:03Z")
I have worked out I can get the data through with series of service restarts that flush the data through the pipe, logstash services followed by filebeat. This is not ideal, but hopefully as the data catches up logstash will be ok to handle this realtime ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2018, 5:01pm UTC](https://discuss.elastic.co/t/elk-server-failed/143179/10 "2018-09-03T17:01:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.