Indices deleted and cluster in read only mode [ES 2.3.4 on AWS]

Elastic Stack Elasticsearch
1

Something happened during the night to my ES cluster (version 2.3.4 on AWS), I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode.

What happened and why? I've lost all data and my cluster entered a read-only mode.

Another guy had my same issue:

2

Are you talking about https://aws.amazon.com/elasticsearch-service/, or https://www.elastic.co/cloud?

3

neither, we launched 2 AWS instaces and setup elasticsearch by hand

4

Check your logs, a cluster shouldn't just switch to red only magically.

5

can you give me an example of log entries to be searched?

6

Look for cluster.blocks.read_only maybe?
Otherwise, put some of it up on gist/pastebin/etc and link here and we can take a look.

7

no entries in logs for that type of search (cluster.blocks.read_only / cluster.blocks / read_only ).

i found sometimes entries like this:

[2016-08-10 13:03:24,148][WARN ][transport.netty ] [assist-production-node-1] exception caught on transport layer [[id: 0x92f09b88, /87.20.245.104:55911 => /172.31.39.183:9300]], closing connection
java.io.IOException: Connessione scaduta
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)