Using 2.2.0, getting "Nul character not allowed"

Elastic Stack Elasticsearch
1

I'm getting this error "Nul character not allowed: wa-2016.06.17" which is causing 2 indexes to remain red after my nodes restarted. I've attempted to delete this one (and one other with the same issue), but elasticsearch complains that the index cannot be found.

Full error from the log below

[2016-06-23 08:01:25,648][WARN ][gateway ] [SJC-PD-LGESMR02] [wa-2016.06.17][1]: failed to list shard for shard_started on node [DZgsMr0VRRmhYVgCcftvRw]
FailedNodeException[Failed node [DZgsMr0VRRmhYVgCcftvRw]]; nested: RemoteTransportException[[SJC-PD-LGESDA02][10.96.24.134:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: IllegalArgumentException[Nul character not allowed: wa-2016.06.17];
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:187)
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$700(TransportNodesAction.java:94)
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$2.handleException(TransportNodesAction.java:160)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:211)
at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:201)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: RemoteTransportException[[SJC-PD-LGESDA02][10.96.24.134:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: IllegalArgumentException[Nul character not allowed: wa-2016.06.17];
Caused by: ElasticsearchException[failed to load started shards]; nested: IllegalArgumentException[Nul character not allowed: wa-2016.06.17];
at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:154)
at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:59)
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:211)
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:207)
at

2

Turns out, there were some control characters in the indexes somehow, this is what it looked like when viewing non-printable characters. (via curling the indices list and piping that through 'cat -v')

w^@a^@-iis-2016.04.07

However, I couldn't get this deleted from the command line. I ended up installing elasticsearch-head. There it saw the indexes (although didn't show the control characters still). I told it to delete the indexes through elasticsearch-head, which it replied as acknowledged false. However it did delete the indexes as requested.