Fyi, 19 hours later, there is no change in the state of the cluster,
and the logs are still bouncing between created and recovering. We
will delete and recreate the missing indices, but it seems like
recovery should have happened by now. What is the expected recovery
time for a shard with a few hundred thousand documents?
Here is the cluster state for reference. The dacbe3f4-ef13-43ba-aa13-
b6457780f145 node had been restarted a few times, and maybe that
caused some shards to get stuck in the relocating state? Still, I'd
expect that at least one shard would have recovered by now.
"routing_table" : {
"indices" : {
"versions" : {
"shards" : {
"0" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 0,
"index" : "versions"
} ],
"1" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 1,
"index" : "versions"
} ],
"2" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 2,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 2,
"index" : "versions"
} ],
"3" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 3,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 3,
"index" : "versions"
} ],
"4" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 4,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 4,
"index" : "versions"
} ],
"5" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 5,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 5,
"index" : "versions"
} ]
}
},
"stats" : {
"shards" : {
"0" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
} ],
"1" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
} ],
"2" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
} ],
"3" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
} ],
"4" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
} ],
"5" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
} ]
}
},
"revisions" : {
"shards" : {
"0" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "revisions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 0,
"index" : "revisions"
} ],
"1" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "revisions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 1,
"index" : "revisions"
} ],
"2" : [ {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 2,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-
b6457780f145",
"shard" : 2,
"index" : "revisions"
} ],
"3" : [ {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 3,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-
b6457780f145",
"shard" : 3,
"index" : "revisions"
} ],
"4" : [ {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 4,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-
b6457780f145",
"shard" : 4,
"index" : "revisions"
} ],
"5" : [ {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 5,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-
b6457780f145",
"shard" : 5,
"index" : "revisions"
} ]
}
}
}
},
"routing_nodes" : {
"unassigned" : [ {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 0,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 1,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 2,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 3,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 4,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 5,
"index" : "versions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 0,
"index" : "revisions"
}, {
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 1,
"index" : "revisions"
} ],
"nodes" : {
"b411c443-ebdd-416a-ac35-cce5af38fa3f" : [ {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 2,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 3,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 4,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "b411c443-ebdd-416a-ac35-cce5af38fa3f",
"relocating_node" : null,
"shard" : 5,
"index" : "revisions"
} ],
"a4eada23-ab66-4ab5-bf20-8de09e25a6a3" : [ {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "versions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "versions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 2,
"index" : "versions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 3,
"index" : "versions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 4,
"index" : "versions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 5,
"index" : "versions"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
}, {
"state" : "STARTED",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 0,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : null,
"shard" : 1,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"shard" : 2,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"shard" : 3,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"shard" : 4,
"index" : "revisions"
}, {
"state" : "RELOCATING",
"primary" : true,
"node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"relocating_node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"shard" : 5,
"index" : "revisions"
} ],
"dacbe3f4-ef13-43ba-aa13-b6457780f145" : [ {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 0,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 1,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 2,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 3,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 4,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : false,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : null,
"shard" : 5,
"index" : "stats"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"shard" : 2,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"shard" : 3,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"shard" : 4,
"index" : "revisions"
}, {
"state" : "INITIALIZING",
"primary" : true,
"node" : "dacbe3f4-ef13-43ba-aa13-b6457780f145",
"relocating_node" : "a4eada23-ab66-4ab5-bf20-8de09e25a6a3",
"shard" : 5,
"index" : "revisions"
} ]
}
On Aug 12, 3:14 am, Shay Banon shay.ba...@elasticsearch.com wrote:
I've pushed a fix for the excessive logging problem.
-shay.banon
On Thu, Aug 12, 2010 at 2:24 AM, Grant Rodgers gra...@gmail.com wrote:
Ok, good to know. I will let it run and see if it finishes
successfully. thanks!
On Aug 11, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:
Thats does not indicate an infinte loop. The recovery process (in this
case,
recovery of a replica shard from the primary shard) is throttled on the
node
level. Basically, in order to not tax the machine the node is running on,
recoveries are throttled (both on the number of concurrent recoveries
allowed, and on the number of concurrent transfers allowed).
A recovery started my moving the shard to RECOVERING state, and then
checks
if it is allowed to perform a recovery. If not, either because the it is
being throttled on the current node, or the node it recovers from, it
restores the previous non recovering state, and retries after an interval
(by default, 100ms).
That process can create a lot of DEBUG level logs (I need to rethink the
logging aspect here), but should finish successfully. It should not
indicate
a problem.
-shay.banon
On Thu, Aug 12, 2010 at 2:10 AM, Grant Rodgers gra...@gmail.com wrote:
We had an incident yesterday where a node ran out of memory while
indexing and the elasticsearch process was killed. Now i'm trying to
recover today and noticing what seems to be an infinite loop in the
recovery process log.
Here's what happens in the log after a restart (filtered to only show
one shard):
[22:59:07,944][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,046][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,047][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,156][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,156][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,266][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,266][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,374][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,374][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,476][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,476][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,586][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,586][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,696][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,696][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,806][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,806][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:08,916][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:08,916][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:09,026][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:09,026][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:09,136][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:09,136][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:09,246][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
[22:59:09,247][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [RECOVERING]->[CREATED], restored after recovery
[22:59:09,356][DEBUG][index.shard.service ] [Joystick] [stats][4]
state: [CREATED]->[RECOVERING]
and many more lines like that, for each shard. ~30000 debug lines from
index.shard.service in less than 3 minutes (before i stopped the
process).
Is this normal? is it a corruption error? it seems to happen
regardless of whether there is a work directory or not.