Elasticsearch upgrade from 6.7 to 7.0

Elastic Stack Elasticsearch
1

Hello,
I recently upgraded to verison 7.0 across my Elasticstack. The following is the errors coming from my elasticsearch.log file. I have tried changing many different aspects of my elasticsearch.yml file but haven't found anything that works. Additionally, I am unable to find anything in the forums that provides a model elasticsearch.yml file for the new upgrade since everything was working fine until I upgraded. Please advise. If there is no solution for this, how do I revert to an earlier version of elastic. Thanks
Best regards,
Megan
using [1] data paths, mounts [[/var (/dev/mapper/systemvg-varlv)]], net usable_space [2.6gb], net total_space [7.9gb], types [xfs]

[2019-04-30T13:44:11,252][INFO ][o.e.e.NodeEnvironment ] [servername] heap size [1007.3mb], compressed ordinary object pointers [true]

[2019-04-30T13:44:11,333][INFO ][o.e.n.Node  ] [servername] node name [servername], node ID [JIQOKjh5SJad0MwxpLG3DA]

[2019-04-30T13:44:11,333][INFO ][o.e.n.Node  ] [servername] version[7.0.0], pid[9995], build[default/rpm/b7e28a7/2019-04-05T22:55:32.697037Z], OS[Linux/3.10.0-514.26.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]

[2019-04-30T13:44:11,334][INFO ][o.e.n.Node  ] [servername] JVM home [/usr/share/elasticsearch/jdk]

[2019-04-30T13:44:11,334][INFO ][o.e.n.Node  ] [servername] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5884753015762719731, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]

[2019-04-30T13:44:13,085][INFO ][o.e.p.PluginsService  ] [servername] loaded module [aggs-matrix-stats]

[2019-04-30T13:44:13,086][INFO ][o.e.p.PluginsService  ] [servername] loaded module [analysis-common]

[2019-04-30T13:44:13,086][INFO ][o.e.p.PluginsService  ] [servername] loaded module [ingest-common]

[2019-04-30T13:44:13,086][INFO ][o.e.p.PluginsService  ] [servername] loaded module [ingest-geoip]

[2019-04-30T13:44:13,086][INFO ][o.e.p.PluginsService  ] [servername] loaded module [ingest-user-agent]

[2019-04-30T13:44:13,086][INFO ][o.e.p.PluginsService  ] [servername] loaded module [lang-expression]

[2019-04-30T13:44:13,087][INFO ][o.e.p.PluginsService  ] [servername] loaded module [lang-mustache]

[2019-04-30T13:44:13,087][INFO ][o.e.p.PluginsService  ] [servername] loaded module [lang-painless]

[2019-04-30T13:44:13,087][INFO ][o.e.p.PluginsService  ] [servername] loaded module [mapper-extras]

[2019-04-30T13:44:13,087][INFO ][o.e.p.PluginsService  ] [servername] loaded module [parent-join]

[2019-04-30T13:44:13,089][INFO ][o.e.p.PluginsService  ] [servername] loaded module [percolator]

[2019-04-30T13:44:13,089][INFO ][o.e.p.PluginsService  ] [servername] loaded module [rank-eval]

[2019-04-30T13:44:13,089][INFO ][o.e.p.PluginsService  ] [servername] loaded module [reindex]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [repository-url]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [transport-netty4]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-ccr]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-core]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-deprecation]

[2019-04-30T13:44:13,090][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-graph]

[2019-04-30T13:44:13,091][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-ilm]

[2019-04-30T13:44:13,091][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-logstash]

[2019-04-30T13:44:13,091][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-ml]

[2019-04-30T13:44:13,091][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-monitoring]

[2019-04-30T13:44:13,093][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-rollup]

[2019-04-30T13:44:13,093][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-security]

[2019-04-30T13:44:13,093][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-sql]

[2019-04-30T13:44:13,093][INFO ][o.e.p.PluginsService  ] [servername] loaded module [x-pack-watcher]

[2019-04-30T13:44:13,094][INFO ][o.e.p.PluginsService  ] [servername] no plugins loaded

[2019-04-30T13:44:17,636][INFO ][o.e.x.s.a.s.FileRolesStore] [servername] parsed [0] roles from file [/etc/elasticsearch/roles.yml]

[2019-04-30T13:44:18,348][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [servername] [controller/10131] [Main.cc@109] controller (64 bit): Version 7.0.0 (Build cdaa022645f38d) Copyright (c) 2019 Elasticsearch BV

[2019-04-30T13:44:18,704][DEBUG][o.e.a.ActionModule  ] [servername] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security

[2019-04-30T13:44:19,306][INFO ][o.e.d.DiscoveryModule ] [servername] using discovery type [zen] and seed hosts providers [settings]

[2019-04-30T13:44:20,411][INFO ][o.e.n.Node  ] [servername] initialized

[2019-04-30T13:44:20,411][INFO ][o.e.n.Node  ] [servername] starting ...

[2019-04-30T13:44:20,536][INFO ][o.e.t.TransportService  ] [servername] publish_address {150.172.106.178:9300}, bound_addresses {150.172.106.178:9300}

[2019-04-30T13:44:20,545][INFO ][o.e.b.BootstrapChecks ] [servername] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2019-04-30T13:44:20,582][INFO ][o.e.c.c.ClusterBootstrapService] [servername] skipping cluster bootstrapping as local node does not match bootstrap requirements: [node-1]
2 
Here is my Elasticsearch.yml file: 
# Use a descriptive name for the node:

#

node.name: node-1

#

# Add custom attributes to the node:

#

#node.attr.rack: r1

#

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /var/lib/elasticsearch

#

# Path to log files:

#

path.logs: /var/log/elasticsearch

#

# ----------------------------------- Memory -----------------------------------

#

# Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

#

# Elasticsearch performs poorly when the system is swapping the memory.

#

# ---------------------------------- Network -----------------------------------

#

# Set the bind address to a specific IP (IPv4 or IPv6):

#

network.host: lxjh452.phibred.com

#

# Set a custom port for HTTP:

#

http.port: 9200

#

# For more information, consult the network module documentation.

#

# --------------------------------- Discovery ----------------------------------

#

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

#

#discovery.seed_hosts: lxjh452.phibred.com

#

# Bootstrap the cluster using an initial set of master-eligible nodes:

#

cluster.initial_master_nodes: node-1

#

# For more information, consult the discovery and cluster formation module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true
3

There don't appear to be any errors in that log file from what I can see?

4

Hello,
I'm so sorry didn't realize the actual error didn't send.

[2019-05-01T08:20:03,945][INFO ][o.e.x.s.a.s.FileRolesStore] [servername] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-05-01T08:20:05,306][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [servername] [controller/23942] [Main.cc@109] controller (64 bit): Version 7.0.0 (Build cdaa022645f38d) Copyright (c) 2019 Elasticsearch BV
[2019-05-01T08:20:06,011][DEBUG][o.e.a.ActionModule       ] [servername] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-05-01T08:20:06,616][INFO ][o.e.d.DiscoveryModule    ] [servername] using discovery type [zen] and seed hosts providers [settings]
[2019-05-01T08:20:07,812][INFO ][o.e.n.Node               ] [servername] initialized
[2019-05-01T08:20:07,813][INFO ][o.e.n.Node               ] [servername] starting ...
[2019-05-01T08:20:07,935][INFO ][o.e.t.TransportService   ] [servername] publish_address {150.172.106.178:9300}, bound_addresses {150.172.106.178:9300}
[2019-05-01T08:20:07,954][INFO ][o.e.b.BootstrapChecks    ] [servername] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-01T08:20:07,981][ERROR][o.e.b.Bootstrap          ] [servername] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-05-01T08:20:07,986][INFO ][o.e.n.Node               ] [servername] stopping ...
[2019-05-01T08:20:07,995][INFO ][o.e.n.Node               ] [servername] stopped
[2019-05-01T08:20:07,996][INFO ][o.e.n.Node               ] [servername] closing ...
[2019-05-01T08:20:08,018][INFO ][o.e.n.Node               ] [servername] closed
[2019-05-01T08:20:08,022][INFO ][o.e.x.m.p.NativeController] [servername] Native controller process has stopped - no new native processes can be started

I hope this is more helpful. Any help or insight is greatly appreciated!

5

You are binding to something other than localhost so the bootstrap checks are in force ...

... and one of the bootstrap checks failed:

The solution is right here in this message: one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured.

6

Hi David,
Thanks for your response. I have noticed this error and hence I went through my elasticserach.yml file to configure these items. I guess my issue is that I am confused as to what hosts/nodes I am supposed to specify. I read the Elastic 7.0 documentation on the matter and tried to adjust my yml file accordingly but I wasn't able to get anything to work. How am I supposed to know what to specify my master node as for instance?

7

I think this page gives a good description of these settings, but if you have read this and are still confused then it'd be useful if you could elaborate on what you've tried and how it didn't work, including any log messages you're seeing.

The elasticsearch.yml file in your original post does seem to have cluster.initial_master_nodes set correctly, so this cannot be the config that corresponds with the log you have shared here.

8

You're right, that's my mistake. I sent the elastic search.yml from when I previously had tried to adjust the settings only to get errors that motivated me to return to my original configuration. The error message from elastic search with the yml that I have posted is as follows:

[ERROR][o.e.x.m.e.l.LocalExporter] [node-1] failed to set monitoring watch [NmTY5EmKQxSJH9gtkF-btQ_elasticsearch_cluster_status]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
    at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:208) ~[elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.blockExceptions(TransportReplicationAction.java:253) ~[elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:762) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:170) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:99) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) [x-pack-security-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:434) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:547) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:253) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:93) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) [x-pack-security-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) [x-pack-security-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.0.0.jar:7.0.0]
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.0.0.jar:7.0.0]
9

This normally means that one or more nodes have (at some point in the past) nearly run out of disk space. This block is applied once a node goes past the flood stage watermark which defaults to 95%. Once it's there, it must be removed manually from any affected indices once you've freed up some disk space.

There's more detail about this, including how to configure it differently and how to remove it, on this page of the manual.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.