Elasticsearch: Unable to start after heavy[20GB] data

Elastic Stack Elasticsearch
1

I am using Elasticsearch, Kibana for monitoring purpose,
Yesterday as Elasticsearch was having 15 GB of indexed
data, it was taking to much load to return result for the
queries by Kibana. It was also showing errors on ES console
so what we decided to do is to stop the instance and restart
it.
But after restarting the elasticsearch, it is staring with proper
started message but after sometime is comes with the following
message

[2016-04-11 15:43:27,890][WARN ][cluster.action.shard     ] [Doctor Druid] [cses
sion-2016.04.11_09.22.19][2] received shard failed for [csession-2016.04.11_09.2
2.19][2], node[0XJiUVb5RvOtD153JuxC0A], [P], v[113], s[INITIALIZING], a[id=CG9to
IZURPqvXt0DP97uWg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-04-11T10
:13:27.347Z], details[failed recovery, failure IndexShardRecoveryException[faile
d recovery]; nested: TranslogException[failed to create new translog file]; nest
ed: FileAlreadyExistsException[D:\ES\elasticsearch\data\elasticsearch\nodes\0\in
dices\csession-2016.04.11_09.22.19\2\translog\translog-2.tlog]; ]], indexUUID [l
m2b-t02SyWO8P_RfBh1qg], message [master {Doctor Druid}{0XJiUVb5RvOtD153JuxC0A}{1
0.76.133.15}{10.76.133.15:9300} marked shard as initializing, but shard is marke
d as failed, resend shard failure], failure [Unknown]
[2016-04-11 15:43:28,048][WARN ][cluster.action.shard     ] [Doctor Druid] [cses
sion-2016.04.11_09.22.19][2] received shard failed for [csession-2016.04.11_09.2
2.19][2], node[0XJiUVb5RvOtD153JuxC0A], [P], v[113], s[INITIALIZING], a[id=CG9to
IZURPqvXt0DP97uWg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-04-11T10
:13:27.347Z], details[failed recovery, failure IndexShardRecoveryException[faile
d recovery]; nested: TranslogException[failed to create new translog file]; nest
ed: FileAlreadyExistsException[D:\ES\elasticsearch\data\elasticsearch\nodes\0\in
dices\csession-2016.04.11_09.22.19\2\translog\translog-2.tlog]; ]], indexUUID [l
m2b-t02SyWO8P_RfBh1qg], message [master {Doctor Druid}{0XJiUVb5RvOtD153JuxC0A}{1
0.76.133.15}{10.76.133.15:9300} marked shard as initializing, but shard is marke
d as failed, resend shard failure], failure [Unknown]
[2016-04-11 15:43:28,317][WARN ][cluster.action.shard     ] [Doctor Druid] [cses
sion-2016.04.11_09.22.19][2] received shard failed for [csession-2016.04.11_09.2
2.19][2], node[0XJiUVb5RvOtD153JuxC0A], [P], v[113], s[INITIALIZING], a[id=CG9to
IZURPqvXt0DP97uWg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-04-11T10
:13:27.347Z], details[failed recovery, failure IndexShardRecoveryException[faile
d recovery]; nested: TranslogException[failed to create new translog file]; nest
ed: FileAlreadyExistsException[D:\ES\elasticsearch\data\elasticsearch\nodes\0\in
dices\csession-2016.04.11_09.22.19\2\translog\translog-2.tlog]; ]], indexUUID [l
m2b-t02SyWO8P_RfBh1qg], message [master {Doctor Druid}{0XJiUVb5RvOtD153JuxC0A}{1
0.76.133.15}{10.76.133.15:9300} marked shard as initializing, but shard is marke
d as failed, resend shard failure], failure [Unknown]

Also, when I start kibana, it displays following error message,

D:\ES\kibana\bin>kibana
  log   [15:42:19.509] [info][status][plugin:kibana] Status changed from uniniti
alized to green - Ready
  log   [15:42:19.536] [info][status][plugin:elasticsearch] Status changed from
uninitialized to yellow - Waiting for Elasticsearch
  log   [15:42:19.549] [info][status][plugin:kbn_vislib_vis_types] Status change
d from uninitialized to green - Ready
  log   [15:42:19.553] [info][status][plugin:markdown_vis] Status changed from u
ninitialized to green - Ready
  log   [15:42:19.562] [info][status][plugin:metric_vis] Status changed from uni
nitialized to green - Ready
  log   [15:42:19.568] [info][status][plugin:spyModes] Status changed from unini
tialized to green - Ready
  log   [15:42:19.572] [info][status][plugin:statusPage] Status changed from uni
nitialized to green - Ready
  log   [15:42:19.575] [info][status][plugin:table_vis] Status changed from unin
itialized to green - Ready
  log   [15:42:19.608] [info][listening] Server running at http://0.0.0.0:5601
  log   [15:42:22.143] [error][status][plugin:elasticsearch] Status changed from
 yellow to red - Elasticsearch is still initializing the kibana index... Trying
again in 2.5 second.
2

Try moving the file that it mentions in the ES log somewhere else, then start ES.

3

Hi @warkolm,
Do you mean translog-2.tlog file as per above log?

4

Yes, don't delete it just move it.

5

Okay @warkolm,

This is a fix or trial an error? I'm asking because I've started another instance for now.
So if its gonna fix, I would need to start topbeat, packetbeat on servers which were stopped.

6

This will stop that error you are seeing.