Shard relocation failed due to filesystem read-only

[xlc]

Hi,

I'm using elasticsearch2.3.2,many warning log detail appered start from yesterday，like
[track_report_idmonitor_hour_17][29] received shard failed for target shard [[track_report_idmonitor_hour_17][29], node[77MLQROxTgWI-FvwLYEghA], relocating [gnpweLOWTAKcLzKKr8sJJQ], [P], v[42919], s[INITIALIZING], a[id=fYgXh-aHQ6ecl2ctN2nGqA, rId=m0uK_m-nSiGtarn9udUecg], expected_shard_size[20728473815]], indexUUID [nC6QvdOPQkS9oqdmUXrl7Q], message [failed to create shard], failure [ElasticsearchException[failed to create shard]; nested: NotSerializableExceptionWrapper[file_system_exception: /ssd1/elasticsearch_data/track_report_production/nodes/0/indices: Read-only file system]; ]

Now，I‘m worried about losing my data,it there some way resolve it safely?By the way, it seems still wirte data successfully.

[Charles.w (Charles Casadei)]

Hi,

How full your disks are ? Is your filesystem writable by the user running elasticsearch ? Are there any warning in elasticsearch logs indicating a high disk usage ?

On a side note:

• to prevent - or at least be warned - of high disk usage, you can configure the disk allocator : https://www.elastic.co/guide/en/elasticsearch/reference/2.3/disk-allocator.html
• I do believe than even with full disks this version of elasticsearch can still accept "write" queries as data is being written in the translog prior being written to disk. For more information on the translog : https://www.elastic.co/guide/en/elasticsearch/reference/2.3/index-modules-translog.html

Best regards,

Charles Casadei.