Elasticsearch version upgrade from 6.6. to 6.8

Elastic Stack Elasticsearch
1

Hello.

I have a server centOS 7 which have Elasticsearch 6.6 and I was trying to upgrade the version to it from 6.6 to 6.8. I followed each step of the guide and all the steps works well. However when I start again the service, I see the version have not changed.

I've installed the version 6.8 from a RPM Repository as show in this page of Elastic. When I did it, the console show me the following:

> # sudo yum install elasticsearch

Loaded plugins: fastestmirror
Repository pgdg96 is listed more than once in the configuration
Repository pgdg96-source is listed more than once in the configuration
Repository pgdg96-updates-testing is listed more than once in the configuration
Repository pgdg96-source-updates-testing is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirror.team-cymru.com
 * epel: pubmirror2.math.uh.edu
 * extras: mirror.genesishosting.com
 * updates: centos.mirrors.tds.net
Package elasticsearch-6.8.12-1.noarch already installed and latest version
Nothing to do

Although the new version is installed, it doesn't seem working when I restart de service.
How can I upgrade the version of the Elasticsearch? Is there an additional step I should have to do?
I really appreciate any suggestion about it. Thanks in advance.

2

Please be more specific what you mean with

it doesn't seem working when I restart de service.

share logfile outputs or journald logs, anything that indicates where this error comes from. Thanks

3

Hi Alexander, thank you for your reply.
I wanted to mean that although the new version was installed, the elasticsearch is still with the version 6.6.
Here's the information requested
Logfile

    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-graph]
    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-ilm]
    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-logstash]
    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-ml]
    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-monitoring]
    [2020-10-06T08:38:48,247][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-rollup]
    [2020-10-06T08:38:48,248][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-security]
    [2020-10-06T08:38:48,248][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-sql]
    [2020-10-06T08:38:48,248][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-upgrade]
    [2020-10-06T08:38:48,248][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-watcher]
    [2020-10-06T08:38:48,248][INFO ][o.e.p.PluginsService     ] [FkhZB6J] no plugins loaded
    [2020-10-06T08:38:55,051][INFO ][o.e.x.s.a.s.FileRolesStore] [FkhZB6J] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
    [2020-10-06T08:38:56,227][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [FkhZB6J] [controller/16247] [Main.cc@114] controller (64 bit): Version 6.8.12 (Build 39bf066b214253) Copyright (c) 2020 Elasticsearch BV
    [2020-10-06T08:38:57,011][DEBUG][o.e.a.ActionModule       ] [FkhZB6J] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
    [2020-10-06T08:38:57,577][INFO ][o.e.d.DiscoveryModule    ] [FkhZB6J] using discovery type [zen] and host providers [settings]
    [2020-10-06T08:38:58,900][INFO ][o.e.n.Node               ] [FkhZB6J] initialized
    [2020-10-06T08:38:59,298][INFO ][o.e.b.BootstrapChecks    ] [FkhZB6J] bound or publishing to a non-loopback address, enforcing bootstrap checks
    [2020-10-06T08:39:02,452][INFO ][o.e.c.s.MasterService    ] [FkhZB6J] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{nKETojSmRyS0ZusAYQwNTQ}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
    [2020-10-06T08:39:02,461][INFO ][o.e.c.s.ClusterApplierService] [FkhZB6J] new_master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{nKETojSmRyS0ZusAYQwNTQ}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{nKETojSmRyS0ZusAYQwNTQ}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
    [2020-10-06T08:39:02,570][INFO ][o.e.h.n.Netty4HttpServerTransport] [FkhZB6J] publish_address {10.128.0.28:9201}, bound_addresses {[::]:9201}
    [2020-10-06T08:39:02,571][INFO ][o.e.n.Node               ] [FkhZB6J] started
    [2020-10-06T08:39:04,959][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [FkhZB6J] Failed to clear cache for realms [[]]
    [2020-10-06T08:39:05,033][INFO ][o.e.l.LicenseService     ] [FkhZB6J] license [ef712658-8d83-4cfd-b8a3-00221f495bf3] mode [basic] - valid
    [2020-10-06T08:39:05,045][INFO ][o.e.g.GatewayService     ] [FkhZB6J] recovered [41] indices into cluster_state
    [2020-10-06T08:39:18,957][INFO ][o.e.c.r.a.AllocationService] [FkhZB6J] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[www.sedeelectronica.flandes.gov.co_gfiles_solicitud][4], [www.sedeelectronica.flandes.gov.co_gfiles_solicitud][0]] ...]).
    [2020-10-06T08:46:06,155][INFO ][o.e.n.Node               ] [FkhZB6J] stopping ...
    [2020-10-06T08:46:06,166][INFO ][o.e.x.w.WatcherService   ] [FkhZB6J] stopping watch service, reason [shutdown initiated]
    [2020-10-06T08:46:06,567][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [FkhZB6J] [controller/16247] [Main.cc@154] ML controller exiting
    [2020-10-06T08:46:06,570][INFO ][o.e.x.m.p.NativeController] [FkhZB6J] Native controller process has stopped - no new native processes can be started
    [2020-10-06T08:46:06,891][INFO ][o.e.n.Node               ] [FkhZB6J] stopped
    [2020-10-06T08:46:06,891][INFO ][o.e.n.Node               ] [FkhZB6J] closing ...
    [2020-10-06T08:46:06,924][INFO ][o.e.n.Node               ] [FkhZB6J] closed
    [2020-10-06T08:46:53,621][INFO ][o.e.e.NodeEnvironment    ] [FkhZB6J] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [192.4gb], net total_space [299.9gb], types [rootfs]
    [2020-10-06T08:46:53,625][INFO ][o.e.e.NodeEnvironment    ] [FkhZB6J] heap size [1007.3mb], compressed ordinary object pointers [true]
    [2020-10-06T08:46:53,927][INFO ][o.e.n.Node               ] [FkhZB6J] node name derived from node ID [FkhZB6JjRMWG-SZfyncngg]; set [node.name] to override
    [2020-10-06T08:46:53,928][INFO ][o.e.n.Node               ] [FkhZB6J] version[6.8.12], pid[16956], build[default/rpm/7a15d2a/2020-08-12T07:27:20.804867Z], OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_262/25.262-b10]
    [2020-10-06T08:46:53,928][INFO ][o.e.n.Node               ] [FkhZB6J] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7725281090071470713, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.max-open-files=true, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
    [2020-10-06T08:46:57,107][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [aggs-matrix-stats]
    [2020-10-06T08:46:57,107][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [analysis-common]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [ingest-common]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [ingest-geoip]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [ingest-user-agent]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [lang-expression]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [lang-mustache]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [lang-painless]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [mapper-extras]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [parent-join]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [percolator]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [rank-eval]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [reindex]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [repository-url]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [transport-netty4]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [tribe]
    [2020-10-06T08:46:57,108][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-ccr]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-core]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-deprecation]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-graph]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-ilm]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-logstash]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-ml]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-monitoring]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-rollup]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-security]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-sql]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-upgrade]
    [2020-10-06T08:46:57,109][INFO ][o.e.p.PluginsService     ] [FkhZB6J] loaded module [x-pack-watcher]
    [2020-10-06T08:46:57,110][INFO ][o.e.p.PluginsService     ] [FkhZB6J] no plugins loaded
    [2020-10-06T08:47:03,873][INFO ][o.e.x.s.a.s.FileRolesStore] [FkhZB6J] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
    [2020-10-06T08:47:05,083][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [FkhZB6J] [controller/17020] [Main.cc@114] controller (64 bit): Version 6.8.12 (Build 39bf066b214253) Copyright (c) 2020 Elasticsearch BV
    [2020-10-06T08:47:05,791][DEBUG][o.e.a.ActionModule       ] [FkhZB6J] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
    [2020-10-06T08:47:06,448][INFO ][o.e.d.DiscoveryModule    ] [FkhZB6J] using discovery type [zen] and host providers [settings]
    [2020-10-06T08:47:07,738][INFO ][o.e.n.Node               ] [FkhZB6J] initialized
    [2020-10-06T08:47:07,739][INFO ][o.e.n.Node               ] [FkhZB6J] starting ...
    [2020-10-06T08:47:08,047][INFO ][o.e.t.TransportService   ] [FkhZB6J] publish_address {10.128.0.28:9300}, bound_addresses {[::]:9300}
    [2020-10-06T08:47:08,182][INFO ][o.e.b.BootstrapChecks    ] [FkhZB6J] bound or publishing to a non-loopback address, enforcing bootstrap checks
    [2020-10-06T08:47:11,347][INFO ][o.e.c.s.MasterService    ] [FkhZB6J] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{IYVkhSeKSLCbJC46LXJZSg}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
    [2020-10-06T08:47:11,355][INFO ][o.e.c.s.ClusterApplierService] [FkhZB6J] new_master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{IYVkhSeKSLCbJC46LXJZSg}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {FkhZB6J}{FkhZB6JjRMWG-SZfyncngg}{IYVkhSeKSLCbJC46LXJZSg}{10.128.0.28}{10.128.0.28:9300}{ml.machine_memory=4142481408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
    [2020-10-06T08:47:11,453][INFO ][o.e.h.n.Netty4HttpServerTransport] [FkhZB6J] publish_address {10.128.0.28:9201}, bound_addresses {[::]:9201}
    [2020-10-06T08:47:11,453][INFO ][o.e.n.Node               ] [FkhZB6J] started
    [2020-10-06T08:47:13,705][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [FkhZB6J] Failed to clear cache for realms [[]]
    [2020-10-06T08:47:13,790][INFO ][o.e.l.LicenseService     ] [FkhZB6J] license [ef712658-8d83-4cfd-b8a3-00221f495bf3] mode [basic] - valid
    [2020-10-06T08:47:13,807][INFO ][o.e.g.GatewayService     ] [FkhZB6J] recovered [41] indices into cluster_state
    [2020-10-06T08:47:27,395][INFO ][o.e.c.r.a.AllocationService] [FkhZB6J] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[www.sedeelectronica.flandes.gov.co_gfiles_solicitud][3]] ...]).
4

There's

version[6.8.12]

In the logs. So it seems to be the latest 6.8 version, no?

1 Like
5

Hello David.

Yes, it is. However, when I sent this request
curl -X GET "localhost:9200/?pretty"
The response show me the following:
{ "name" : "FkhZB6J", "cluster_name" : "elasticsearch", "cluster_uuid" : "8tvZeiBUT1e78pMwyJAygQ", "version" : { "number" : "6.6.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "a9861f4", "build_date" : "2019-01-24T11:27:09.439740Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

6

It's probably because your instance is running at

10.128.0.28:9201

According to the logs.

1 Like
7

It seems like you have 2 instance of elasticsearch running.
one with version 6.6 running at localhost:9200
and one with version 6.8 running at 10.128.0.28:9201

8

Yes, you're right. I have two instance in my server: localhost:9200 is running the 6.6 and localhost:9201 is running the version 6.8.12.
I though doing the upgrade process my first instance would be updated but it wasn't. Now, How can I remove my first instance? I just need the one upgraded

I was trying to stop the first instance with the following but it stopped the second one:
sudo systemctl stop elasticsearch
All commands with systemctl that I do is are applied to the second instance.

I notice the I have 2 files in /usr/lib/systemd/system/ called elasticsearch.service and elasticsearch.service.rpmnew. In adition, I see two script files in /etc/init.d/ called elasticsearch and elasticsearch.rpmnew

Thanks for your help in advance.

9

As I never encounter this issue before, I am not able to give any comment on this.

Could @spinscale @dadoonet could shed some light onto this issue?

1 Like
10

I think this boils down if the steps to perform an upgrade have been executed in the right order - this one here seems as if someone started an upgrade without verifying if the Elasticsearch process was properly shutdown.

Have all the steps listed at https://www.elastic.co/guide/en/elasticsearch/reference/7.9/rolling-upgrades.html been executed in the right order (including verification)?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.