# Google Cloud Storage Repository Plugin - "state" : "PARTIAL"

**URL:** <https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225>\
**Category:** Elasticsearch\
**Created:** [April 8, 2018, 7:14pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225 "2018-04-08T19:14:22Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 8, 2018, 7:14pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/1 "2018-04-08T19:14:23Z")

</div>

Hello,

I'm using Elastic (6.2.2) Stack w/ [Google Cloud Storage Repository Plugin | Elasticsearch Plugins and Integrations](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs.html).

I keep getting "PARTIAL" instead of "SUCCESS" state of my snapshot:

```
# time curl --silent --header 'Content-Type: application/json' --request PUT "$ELASTICSEARCH_URI/_snapshot/repository-gcs/X?wait_for_completion=true&pretty" --data '{"indices":"X"}'
{
  "snapshot" : {
    "snapshot" : "X",
    "uuid" : "NuP6yLBYQoq0ZsChRBS04g",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [
      "X"
    ],
    "include_global_state" : true,
    "state" : "PARTIAL",
    "start_time" : "2018-04-08T19:02:59.717Z",
    "start_time_in_millis" : 1523214179717,
    "end_time" : "2018-04-08T19:03:01.184Z",
    "end_time_in_millis" : 1523214181184,
    "duration_in_millis" : 1467,
    "failures" : [
      {
        "index" : "X",
        "index_uuid" : "X",
        "shard_id" : 3,
        "reason" : "RepositoryMissingException[[repository-gcs] missing]",
        "node_id" : "LPVtuXogTkiB7sk8ibnI0w",
        "status" : "INTERNAL_SERVER_ERROR"
      }
    ],
    "shards" : {
      "total" : 5,
      "failed" : 1,
      "successful" : 4
    }
  }
}

real	0m3.194s
user	0m0.004s
sys	0m0.004s
# 

```

elasticsearch log (master):

```
esm1 | [2018-04-08T19:03:00,391][INFO][o.e.s.SnapshotsService] [esm1] snapshot [repository-gcs:X/NuP6yLBYQoq0ZsChRBS04g] started
esm1 | [2018-04-08T19:03:02,606][INFO][o.e.s.SnapshotsService] [esm1] snapshot [repository-gcs:X/NuP6yLBYQoq0ZsChRBS04g] completed with state [PARTIAL]

```

elasticsearch log (data):

```
esd2 | [2018-04-08T19:03:00,413][WARN][o.e.s.SnapshotShardsService] [esd2] [[X][3]][repository-gcs:X/NuP6yLBYQoq0ZsChRBS04g] failed to snapshot shard
esd2 | org.elasticsearch.repositories.RepositoryMissingException: [repository-gcs] missing
esd2 | at org.elasticsearch.repositories.RepositoriesService.repository(RepositoriesService.java:335) ~[elasticsearch-6.2.2.jar:6.2.2]
esd2 | at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:408) ~[elasticsearch-6.2.2.jar:6.2.2]
esd2 | at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:98) ~[elasticsearch-6.2.2.jar:6.2.2]
esd2 | at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:355) [elasticsearch-6.2.2.jar:6.2.2]
esd2 | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2]
esd2 | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2]
esd2 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
esd2 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
esd2 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

```

it works for some, however not for all(

Please advise.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 8, 2018, 7:44pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/2 "2018-04-08T19:44:07Z")

</div>

What is the output of

```
GET /_cat/plugins?v
```

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 8, 2018, 11:17pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/3 "2018-04-08T23:17:50Z")

</div>

```
# curl --silent --request GET $ELASTICSEARCH_URI/_cat/plugins?v
name component version
esc3 ingest-geoip 6.2.2
esc3 ingest-user-agent 6.2.2
esc3 repository-gcs 6.2.2
esc3 x-pack-core 6.2.2
esc3 x-pack-deprecation 6.2.2
esc3 x-pack-graph 6.2.2
esc3 x-pack-logstash 6.2.2
esc3 x-pack-ml 6.2.2
esc3 x-pack-monitoring 6.2.2
esc3 x-pack-security 6.2.2
esc3 x-pack-upgrade 6.2.2
esc3 x-pack-watcher 6.2.2
esd2 ingest-geoip 6.2.2
esd2 ingest-user-agent 6.2.2
esd2 repository-gcs 6.2.2
esd2 x-pack-core 6.2.2
esd2 x-pack-deprecation 6.2.2
esd2 x-pack-graph 6.2.2
esd2 x-pack-logstash 6.2.2
esd2 x-pack-ml 6.2.2
esd2 x-pack-monitoring 6.2.2
esd2 x-pack-security 6.2.2
esd2 x-pack-upgrade 6.2.2
esd2 x-pack-watcher 6.2.2
esm1 ingest-geoip 6.2.2
esm1 ingest-user-agent 6.2.2
esm1 repository-gcs 6.2.2
esm1 x-pack-core 6.2.2
esm1 x-pack-deprecation 6.2.2
esm1 x-pack-graph 6.2.2
esm1 x-pack-logstash 6.2.2
esm1 x-pack-ml 6.2.2
esm1 x-pack-monitoring 6.2.2
esm1 x-pack-security 6.2.2
esm1 x-pack-upgrade 6.2.2
esm1 x-pack-watcher 6.2.2
esd5 ingest-geoip 6.2.2
esd5 ingest-user-agent 6.2.2
esd5 repository-gcs 6.2.2
esd5 x-pack-core 6.2.2
esd5 x-pack-deprecation 6.2.2
esd5 x-pack-graph 6.2.2
esd5 x-pack-logstash 6.2.2
esd5 x-pack-ml 6.2.2
esd5 x-pack-monitoring 6.2.2
esd5 x-pack-security 6.2.2
esd5 x-pack-upgrade 6.2.2
esd5 x-pack-watcher 6.2.2
esm3 ingest-geoip 6.2.2
esm3 ingest-user-agent 6.2.2
esm3 repository-gcs 6.2.2
esm3 x-pack-core 6.2.2
esm3 x-pack-deprecation 6.2.2
esm3 x-pack-graph 6.2.2
esm3 x-pack-logstash 6.2.2
esm3 x-pack-ml 6.2.2
esm3 x-pack-monitoring 6.2.2
esm3 x-pack-security 6.2.2
esm3 x-pack-upgrade 6.2.2
esm3 x-pack-watcher 6.2.2
esc1 ingest-geoip 6.2.2
esc1 ingest-user-agent 6.2.2
esc1 repository-gcs 6.2.2
esc1 x-pack-core 6.2.2
esc1 x-pack-deprecation 6.2.2
esc1 x-pack-graph 6.2.2
esc1 x-pack-logstash 6.2.2
esc1 x-pack-ml 6.2.2
esc1 x-pack-monitoring 6.2.2
esc1 x-pack-security 6.2.2
esc1 x-pack-upgrade 6.2.2
esc1 x-pack-watcher 6.2.2
esd1 ingest-geoip 6.2.2
esd1 ingest-user-agent 6.2.2
esd1 repository-gcs 6.2.2
esd1 x-pack-core 6.2.2
esd1 x-pack-deprecation 6.2.2
esd1 x-pack-graph 6.2.2
esd1 x-pack-logstash 6.2.2
esd1 x-pack-ml 6.2.2
esd1 x-pack-monitoring 6.2.2
esd1 x-pack-security 6.2.2
esd1 x-pack-upgrade 6.2.2
esd1 x-pack-watcher 6.2.2
elasticsearch ingest-geoip 6.2.2
elasticsearch ingest-user-agent 6.2.2
elasticsearch repository-gcs 6.2.2
elasticsearch x-pack-core 6.2.2
elasticsearch x-pack-deprecation 6.2.2
elasticsearch x-pack-graph 6.2.2
elasticsearch x-pack-logstash 6.2.2
elasticsearch x-pack-ml 6.2.2
elasticsearch x-pack-monitoring 6.2.2
elasticsearch x-pack-security 6.2.2
elasticsearch x-pack-upgrade 6.2.2
elasticsearch x-pack-watcher 6.2.2
esd4 ingest-geoip 6.2.2
esd4 ingest-user-agent 6.2.2
esd4 repository-gcs 6.2.2
esd4 x-pack-core 6.2.2
esd4 x-pack-deprecation 6.2.2
esd4 x-pack-graph 6.2.2
esd4 x-pack-logstash 6.2.2
esd4 x-pack-ml 6.2.2
esd4 x-pack-monitoring 6.2.2
esd4 x-pack-security 6.2.2
esd4 x-pack-upgrade 6.2.2
esd4 x-pack-watcher 6.2.2
esd3 ingest-geoip 6.2.2
esd3 ingest-user-agent 6.2.2
esd3 repository-gcs 6.2.2
esd3 x-pack-core 6.2.2
esd3 x-pack-deprecation 6.2.2
esd3 x-pack-graph 6.2.2
esd3 x-pack-logstash 6.2.2
esd3 x-pack-ml 6.2.2
esd3 x-pack-monitoring 6.2.2
esd3 x-pack-security 6.2.2
esd3 x-pack-upgrade 6.2.2
esd3 x-pack-watcher 6.2.2
esm2 ingest-geoip 6.2.2
esm2 ingest-user-agent 6.2.2
esm2 repository-gcs 6.2.2
esm2 x-pack-core 6.2.2
esm2 x-pack-deprecation 6.2.2
esm2 x-pack-graph 6.2.2
esm2 x-pack-logstash 6.2.2
esm2 x-pack-ml 6.2.2
esm2 x-pack-monitoring 6.2.2
esm2 x-pack-security 6.2.2
esm2 x-pack-upgrade 6.2.2
esm2 x-pack-watcher 6.2.2
#
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 9, 2018, 6:40am UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/4 "2018-04-09T06:40:01Z")

</div>

Is there anything else in logs (in all logs)?

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 9, 2018, 12:36pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/5 "2018-04-09T12:36:20Z")

</div>

I tail all nodes logs, then execute `curl` command (logs that I included in first post is what happens after). Is there anything else I can provide to help you troubleshoot?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 9, 2018, 1:08pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/6 "2018-04-09T13:08:39Z")

</div>

I'm running out of ideas. May be @Igor_Motov or @tanguy have an idea.  
Looks like one node is not aware of the repository which I would only explain with a split brain and a cluster state not correctly sent to all nodes but I think that in that case logs would tell that.

Could you remove the repository? Create it again?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 9, 2018, 2:36pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/7 "2018-04-09T14:36:22Z")

</div>

@alexus do all nodes have the same settings (including secure settings)? Did you restart nodes after updating these settings?

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 9, 2018, 3:08pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/8 "2018-04-09T15:08:36Z")

</div>

@Igor_Motov Apart for `node.*` settings (as I have dedicated master, data, ingest and coordinate node), as well memory settings, all nodes are exactly identical.

Which security settings or changes are you referring too?

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 9, 2018, 4:05pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/9 "2018-04-09T16:05:05Z")

</div>

Per [Repository Verification](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_repository_verification):

> When a repository is registered, it’s immediately verified on all master and data nodes to make sure that it is functional on all nodes currently present in the cluster. The verify parameter can be used to explicitly disable the repository verification when registering or updating a repository:

... needless to say I did not use `?verify=false` when registered my repository.

However after deleting my repository and re-registering it again, followed by creating new snapshot with new state `SUCCESS`!

Thanks!

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [April 9, 2018, 4:33pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/10 "2018-04-09T16:33:30Z")

</div>

I was referring to GCS settings. The problem that you describe typically occur when node settings on nodes are getting out of sync or a plugin is not installed on some nodes during the repository registration.

---

<div class="post-metadata">

**Author:** ![alexus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexus/32/12696_2.png) [@alexus](https://discuss.elastic.co/u/alexus)\
**Post date:** [April 9, 2018, 4:40pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/11 "2018-04-09T16:40:53Z")

</div>

@Igor_Motov I see, in my case I use `Dockerfile` (where I integrate GCS settings). I don't see how or why would one of the nodes, somehow get out of sync or loose plugin that was installed into image, but it looks like somehow it did get out of sync as per David's requirements I deleted and re-registered repository and it seems to address my issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2018, 4:41pm UTC](https://discuss.elastic.co/t/google-cloud-storage-repository-plugin-state-partial/127225/12 "2018-05-07T16:41:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.