# General Guidance on Updating ELK with minimum Fuss

**URL:** <https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828>\
**Category:** Elasticsearch\
**Created:** [May 15, 2020, 1:10pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828 "2020-05-15T13:10:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [May 15, 2020, 1:10pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/1 "2020-05-15T13:10:17Z")

</div>

I might be missing something but ELK updates (even minor versions like 7.6.2 to 7.7) can be a real pain with much downtime. Is there a guide which explains how to upgrade?

We generally download the latest version, copy `data` and `config` directories across and start the services.

Today we updated ElasticSearch and Kibana from 7.6.2 to 7.7 and ended up with all our indices red.

It seems apparent that ES is doing something to the indices as it's log is running full and the list of red indices is decreasing.

We are seeing throttling in allocation: although we only have one node and every index has 1 shard and no replicas. Kibana is running but the logs indicate it's hitting a not-quite-ready elasticsearch.

Was there some update process we missed which would save us downtime?

There is NOTHING in the elasticsearch log about why the node is red.

```auto
[2020-05-15T14:34:09,182][WARN][r.suppressed] [ecom-repository01] path: /.kibana/_doc/space%3Adefault, params: {index=.kibana, id=space:default}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.7.0.jar:7.7.0]

```

```auto
[2020-05-15T15:04:11,241][WARN][o.e.x.m.e.l.LocalExporter] [ecom-repository01] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-7-2020.05.15][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-7-2020.05.15][0]] containing [414] requests]]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.7.0.jar:7.7.0]

```

```auto
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:196) ~[elasticsearch-7.7.0.jar:7.7.0]

```

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [May 15, 2020, 2:03pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/2 "2020-05-15T14:03:31Z")

</div>

Hi @cawoodm,

I run ES on Debian Linux and have the configuration Puppet managed. The ES service is not restarted on config change.

I can upgrade the ES package and do a [rolling restart](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-rolling) of the whole cluster with no down time.

I have also [shard allocation awareness](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html) configured which lets me restart groups of nodes instead of one node at a time when doing a rolling restart.

I never move the data.

P.S. I only now read that you only have one node... That will probably cause at least brief RED at upgrades...

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 15, 2020, 2:15pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/3 "2020-05-15T14:15:02Z")

</div>

> [@cawoodm](#):
>
> Is there a guide which explains how to upgrade?

Yes, a step-by-step guide through the proper procedure for a zero-downtime upgrade is in the [reference documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html).

> [@cawoodm](#):
>
> We generally download the latest version, copy `data` and `config` directories across and start the services.

That sounds like a mistake, you don't need to move or copy the `data` directory to upgrade.

> [@cawoodm](#):
>
> although we only have one node

Oh, in which case you cannot avoid downtime. The minimum-fuss method requires at least three nodes.

> [@cawoodm](#):
>
> There is NOTHING in the elasticsearch log about why the node is red.

Here's a blog post that might help:

> **[RED Elasticsearch Cluster? Panic no longer](https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer)**
>
> Your cluster is RED - you have unassigned shards and your cluster is not fully operational. Introducing a new Elasticsearch API which will save the day.

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [May 17, 2020, 9:10am UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/4 "2020-05-17T09:10:35Z")

</div>

OK, so there is no official guide on how to upgrade if we only have one node?

Our strategy so far was to:

- Stop the service running in directory`elasticsearch.v.old'
- Copy `config` and `data` across to directory`elasticsearch.v.new'
- Fire up `elasticsearch.v.new' and see what happens
- In case of problems, simply go back to running`elasticsearch.v.old'

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 17, 2020, 9:33am UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/5 "2020-05-17T09:33:35Z")

</div>

Yes, there's a documented procedure for [upgrading by restarting the whole cluster too](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html). It involves downtime, however, there's no way around that if you only have one node.

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [May 17, 2020, 7:02pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/6 "2020-05-17T19:02:43Z")

</div>

That is one painful upgrade process you got there son.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2020, 7:02pm UTC](https://discuss.elastic.co/t/general-guidance-on-updating-elk-with-minimum-fuss/232828/7 "2020-06-14T19:02:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.