Failure while trying to create a snapshot

[Kennethtruyers]

Whenever I try to create a snapshot of a certain set of indices, I get the error ElasticsearchParseException[missing or invalid written_by [null]]

I'm trying to create a snapshot using the Azure plugin. On some index-sets the snapshot seems to work, but on other it fails.

Here's an example of one where it fails. It fails on multiple shards. I've tried reindexing the data (dropping and creating the indices from scratch), but the error stays the same:

Request:

PUT http://<elastic_url>/_snapshot/azurerepo/812d10b89bb84de3a2f5622a2417cdda?wait_for_completion=true 
{
    "indices":"*qa-master*"
}

Response:

HTTP/1.1 200 OK

{
	"snapshot": {
		"snapshot": "812d10b89bb84de3a2f5622a2417cdda",
		"uuid": "PKG5m6vORYeL22GzXp7COg",
		"version_id": 5050099,
		"version": "5.5.0",
		"indices": ["cms-qa-master-entity-authority",
		"cms-qa-master-entity-carpark",
		"cms-qa-master-entity-region",
		"cms-qa-master-entity-country",
		"cms-qa-master-entity-zone",
		"cms-qa-master-entity-bundle",
		"cms-qa-master-entity-osp",
		"cms-qa-master-entity-dataorigin"],
		"state": "PARTIAL",
		"start_time": "2017-08-30T09:00:47.724Z",
		"start_time_in_millis": 1504083647724,
		"end_time": "2017-08-30T09:01:11.846Z",
		"end_time_in_millis": 1504083671846,
		"duration_in_millis": 24122,
		"failures": [{
			"index": "cms-qa-master-entity-country",
			"index_uuid": "cms-qa-master-entity-country",
			"shard_id": 0,
			"reason": "IndexShardSnapshotFailedException[ElasticsearchParseException[missing or invalid written_by [null]]]; nested: ElasticsearchParseException[missing or invalid written_by [null]]; ",
			"node_id": "kdDt4y_dQaKQWbsNWcHbaw",
			"status": "INTERNAL_SERVER_ERROR"
		},
		{
			"index": "cms-qa-master-entity-bundle",
			"index_uuid": "cms-qa-master-entity-bundle",
			"shard_id": 0,
			"reason": "IndexShardSnapshotFailedException[ElasticsearchParseException[missing or invalid written_by [null]]]; nested: ElasticsearchParseException[missing or invalid written_by [null]]; ",
			"node_id": "pu-r1kTQTOGqegAF6Agv-A",
			"status": "INTERNAL_SERVER_ERROR"
		}],
		"shards": {
			"total": 8,
			"failed": 2,
			"successful": 6
		}
	}
}

I've tried recreating the indices with a different number of shards (same documents), but the problem remains the same (just more shards failing)

Note: the snapshot always fails on the same two indices (cms-qa-master-entity-country and cms-qa-master-entity-bundle).
Note 2: these indices contain very few documents (between 1-5).
Note 3: I did the same for other index sets (eg: *qa-other*) and that created the snapshots fine. These other index sets have more or less the same amount of documents and the *-entity-country and *-entity-bundle indexes are in fact identical, in terms of mapping, document count and documents.

[spinscale (Alexander Reelsen)]

Did you spot anything in the log files of the nodes which contain these shards? Also, I think it makes sense to actually open an issue for this one in the github repo. Even if this turns out not to be a bug, the exception is not helping to resolve the issue.

So, please go ahead and open an issue. Thanks a lot!