Cannot start ES inside LXC using _site_ (error shows memory issue)

Hi, I can't seem to start ES inside my Centos 7 container. I am using mac --> multipass --> LXD --> LXC centos7.

So I can start ES with no errors if I remove site in my yml config like this:

network.host: ["_local_"]

But if I use

network.host: ["_local_", "_site_"]

It gives me this error:

[root@master-1 ~]# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

See screenshot from: journalctl -xe

So this is a memory related issue if I use site?

Welcome to our community! :smiley:
Please don't post pictures of text or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

What do your Elasticsearch logs show?

I am sorry about that. Please see this pastebin: [root@master-1 elasticsearch]# rm * -f[root@master-1 elasticsearch]# vi /etc/e - Pastebin.com

I deleted the logs inside /var/log/elasticsearch then add site in yml then restart ES those logs are generated see pastebin.

These are also what I got from journalctl -xe


Jul 05 04:42:43 master-1 systemd[1]: Stopping Elasticsearch...

-- Subject: Unit elasticsearch.service has begun shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit elasticsearch.service has begun shutting down.

Jul 05 04:42:43 master-1 systemd[1]: Stopped Elasticsearch.

-- Subject: Unit elasticsearch.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit elasticsearch.service has finished shutting down.

Jul 05 04:42:43 master-1 systemd[1]: Starting Elasticsearch...

-- Subject: Unit elasticsearch.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit elasticsearch.service has begun starting up.

Jul 05 04:42:44 master-1 elasticsearch[8721]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be r

emoved in a future release.

Jul 05 04:43:18 master-1 elasticsearch[8721]: uncaught exception in thread [main]

Jul 05 04:43:18 master-1 elasticsearch[8721]: BindTransportException[Failed to resolve host [_local_, _site]]; nested: UnknownHostException[_site: Name or s

ervice not known];

Jul 05 04:43:18 master-1 elasticsearch[8721]: Likely root cause: java.net.UnknownHostException: _site: Name or service not known

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1499)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:849)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1489)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1348)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1282)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.common.network.NetworkService.resolveInternal(NetworkService.java:248)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:198)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.common.network.NetworkService.resolveBindHostAddresses(NetworkService.java:114)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:339)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:135)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transp

ort.java:81)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4Serve

rTransport.java:43)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.transport.TransportService.doStart(TransportService.java:231)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.node.Node.start(Node.java:697)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.cli.Command.main(Command.java:90)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)

Jul 05 04:43:18 master-1 elasticsearch[8721]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

Jul 05 04:43:18 master-1 elasticsearch[8721]: For complete error details, refer to the log at /var/log/elasticsearch/playground.log

Jul 05 04:43:18 master-1 systemd[1]: **elasticsearch.service: main process exited, code=exited, status=1/FAILURE**

Jul 05 04:43:18 master-1 systemd[1]: **Failed to start Elasticsearch.**

-- Subject: Unit elasticsearch.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit elasticsearch.service has failed.

--

-- The result is failed.

Jul 05 04:43:18 master-1 systemd[1]: **Unit elasticsearch.service entered failed state.**

Jul 05 04:43:18 master-1 systemd[1]: **elasticsearch.service failed.**


What version are you running?

Also the correct setting is _local_ or _site_, they do not need quotes around them.

I tried quotes and without quotes:

#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: [_local_, _site_]
#
# Set a custom port for HTTP:
#
http.port: 9200

Version is:

[root@master-1 bin]# ./elasticsearch --version
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 7.6.0, Build: default/rpm/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z, JVM: 13.0.2
[root@master-1 bin]# 

From journalctl -xe

-- 
-- Unit elasticsearch.service has begun starting up.
Jul 05 05:27:27 master-1 elasticsearch[9322]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be r
emoved in a future release.
Jul 05 05:28:00 master-1 elasticsearch[9322]: ERROR: [1] bootstrap checks failed
Jul 05 05:28:00 master-1 elasticsearch[9322]: [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Jul 05 05:28:00 master-1 elasticsearch[9322]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/playground.log
Jul 05 05:28:00 master-1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Jul 05 05:28:00 master-1 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed

it says: " max virtual memory areas vm.max_map_count [65530]"

So that is something to do with "site"? I remove "site" and it works just fine.

Ok I think it'd be useful if you could please post your entire elasticsearch.yml.

Please upgrade, that version is nearly EOL and there has been a number of product improvements made. Latest is 7.13.

Now I am using 7.13.2

Here is the yml file:

[root@master-1 elasticsearch]# rpm -qa | grep elas
elasticsearch-7.13.2-1.x86_64

[root@master-1 elasticsearch]# clear

[root@master-1 elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: playground
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: [_local_]
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.41.55.114"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["master-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
node.master: true
node.data: false
node.ingest: true
node.ml: false
~                 
[root@master-1 elasticsearch]# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[root@master-1 elasticsearch]# ^C
[root@master-1 elasticsearch]# ^C
[root@master-1 elasticsearch]# ^C

Now it's giving a different error from journalctl -xe

-
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: in 'reader', line 87, column 1:
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ~
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: could not find expected ':'
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: in 'reader', line 88, column 1:
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: at [Source: (sun.nio.ch.ChannelInputStream); line: 86, column: 15]]; nested: ScannerException[while scann
ing a simple key
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: in 'reader', line 87, column 1:
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ~
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: could not find expected ':'
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: in 'reader', line 88, column 1:
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: ];
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1102)
Jul 05 06:06:53 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1075)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:72)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:89)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:80)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.cli.Command.main(Command.java:79)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a si
mple key
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: in 'reader', line 87, column 1:
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ~
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: could not find expected ':'
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: in 'reader', line 88, column 1:
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at [Source: (sun.nio.ch.ChannelInputStream); line: 86, column: 15]
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.jav
a:27)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:359)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:41)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:634)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:609)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings.access$400(Settings.java:71)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1098)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ... 9 more
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: Caused by: while scanning a simple key
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: in 'reader', line 87, column 1:
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ~
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: could not find expected ':'
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: in 'reader', line 88, column 1:
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ^
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:464)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:278)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:226)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:355)
Jul 05 06:06:54 master-1 systemd-entrypoint[5037]: ... 14 more
Jul 05 06:06:54 master-1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 05 06:06:54 master-1 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit elasticsearch.service has failed.
-- 
-- The result is failed.
Jul 05 06:06:54 master-1 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 05 06:06:54 master-1 systemd[1]: elasticsearch.s

If you connect to anything apart from localhost, which _site_ does, this triggers Elasticsearch to enforce bootstrap checks and this is what is failing according to your error messages. Have a look at the docs and the guidance around the different bootstrap checks.

Yeah something to do with bootstrap check. Now I got this error using the latest version of ES

-- Unit elasticsearch.service has begun starting up.
Jul 05 23:34:47 data-1 systemd-entrypoint[2193]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Jul 05 23:34:47 data-1 systemd-entrypoint[2193]: bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Jul 05 23:34:47 data-1 systemd-entrypoint[2193]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/playground.log
Jul 05 23:34:47 data-1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Jul 05 23:34:47 data-1 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Did you check all of the bootstrap checks?

Yes I checked this bootstrap checks: A complete overview of bootstrap checks in Elasticsearch

But I tried those especially the "sysctl vm.max_map_count 262144"

But no luck. Error for that:

[root@data-2 ~]# sysctl vm.max_map_count 262144
vm.max_map_count = 65530
sysctl: cannot stat /proc/sys/262144: No such file or directory

I fixed it! Like I mentioned from my original post, I am using macOS --> Ubuntu using multipass with LXD inside it --> LXC/container centOS.

I was setting up the max_map_count inside LXC container that's why I cannot change it "permission denied.

[root@data-2 ~]# echo 262144 > /proc/sys/vm/max_map_count
bash: /proc/sys/vm/max_map_count: Permission denied

It needs to be done from host (LXD not the LXC/container) from this reference.

From LXD/host

ubuntu@elk:~$ sudo sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

From LXC/container:

[root@data-2 ~]# sysctl vm.max_map_count
vm.max_map_count = 262144

[root@data-2 ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch

[root@data-2 ~]# grep network.host /etc/elasticsearch/elasticsearch.yml 
network.host: [_local_, _site_]

But any ideas why "site" has something to do with "vm.max_map_count"?

max virtual memory areas vm.max_map_count [65530] 

I set it from 65530 to 262144 and that fixed it!