# Disk got full and the shards never seemed to be recovered?
**URL:** https://discuss.elastic.co/t/disk-got-full-and-the-shards-never-seemed-to-be-recovered/90222
**Category:** Elasticsearch
**Created:** [June 21, 2017, 8:09am UTC](https://discuss.elastic.co/t/disk-got-full-and-the-shards-never-seemed-to-be-recovered/90222 "2017-06-21T08:09:03Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@blommis](https://discuss.elastic.co/u/blommis)
#### Post date: [June 21, 2017, 8:09am UTC](https://discuss.elastic.co/t/disk-got-full-and-the-shards-never-seemed-to-be-recovered/90222/1 "2017-06-21T08:09:03Z")
So, WHILE we were indexing, which we do every 30 min, the disk got full. The bulkprocessor (java) got hanging and never quit the indexing. Or thew an exception.
The disk got fixed, but the index process still seemed to be locked.
Elasticsearch logged NoShardAvailableActionException.
The bulkprocessor remained locked and the indices were stuck at red.
What I did?
I deleted the indices and restarted the service that indexes elasticsearch.
Is there any better way to handle this?
$curl localhost:12400/\_cluster/health | python -mjson.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
129 389 129 389 0 0 70650 0 --:--:-- --:--:-- --:--:-- 379k
{
"active\_primary\_shards": 0,
"active\_shards": 0,
"active\_shards\_percent\_as\_number": 0.0,
"cluster\_name": "oppdragsregister",
"delayed\_unassigned\_shards": 0,
"initializing\_shards": 0,
"number\_of\_data\_nodes": 2,
"number\_of\_in\_flight\_fetch": 0,
"number\_of\_nodes": 4,
"number\_of\_pending\_tasks": 0,
"relocating\_shards": 0,
"status": "red",
"task\_max\_waiting\_in\_queue\_millis": 0,
"timed\_out": false,
"unassigned\_shards": 20
}
$ curl localhost:12400/\_cluster/allocation/explain | python -mjson.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
106 746 106 746 0 0 10917 0 --:--:-- --:--:-- --:--:-- 11134
{
"allocate\_explanation": "can allocate the shard",
"can\_allocate": "yes",
"current\_state": "unassigned",
"index": "blabla:2017-06-21t07:07:58.601",
"node\_allocation\_decisions": [
{
"node\_decision": "yes",
"node\_id": "NbWEPxmBQn-GVvQcIlZMDg",
"node\_name": "blabla-data",
"transport\_address": "10...:12500",
"weight\_ranking": 1
},
{
"node\_decision": "yes",
"node\_id": "zr6vKM7-RJmTYTiqjtQ7vw",
"node\_name": "blabla-data",
"transport\_address": "10...:12500",
"weight\_ranking": 2
}
],
"primary": true,
"shard": 1,
"target\_node": {
"id": "NbWEPxmBQn-GVvQcIlZMDg",
"name": "blabla-data",
"transport\_address": "...:12500"
},
"unassigned\_info": {
"at": "2017-06-21T05:07:58.612Z",
"last\_allocation\_status": "no",
"reason": "INDEX\_CREATED"
}
}
> [2017-06-21T09:52:27,820][DEBUG][o.e.a.s.TransportSearchAction] [nodedata] All shards failed for phase: [query]
> org.elasticsearch.action.NoShardAvailableActionException: null
> at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:115) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:154) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1488) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1445) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1329) [elasticsearch-5.2.0.jar:5.2.0]
> at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.2.0.jar:5.2.0]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
---
### Author:  [@Xavy](https://discuss.elastic.co/u/Xavy)
#### Post date: [June 21, 2017, 8:19am UTC](https://discuss.elastic.co/t/disk-got-full-and-the-shards-never-seemed-to-be-recovered/90222/2 "2017-06-21T08:19:09Z")
Hello:
I guess so, but, could you confirm the same message if you make a reroute API call, with rety\_failed?
[https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 19, 2017, 8:19am UTC](https://discuss.elastic.co/t/disk-got-full-and-the-shards-never-seemed-to-be-recovered/90222/3 "2017-07-19T08:19:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.