Bypassing failing bootstrap checks

Elastic Stack Elasticsearch
1

I am trying to run 3 elasticsearch masters on a single node cluster but am having issues regarding the virtual memory setting. Everything seems to be running fine locally if I create the cluster/masters after first using sysctl to increase the virtual memory but the issue is that in production we do not run in privileged mode/allow modification to the virtual memory so I cannot add the sysctl procedure to create my elasticsearch cluster. I tried to get the cluster to start up and run locally without the sysctl beforehand and got the following message:

[2017-10-30T15:46:12,338][INFO ][o.e.b.BootstrapChecks    ] [es-0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Although it isn't recommended, I also tried by disabling the bootstrap checks by adding the discovery.type: single-node config to my elasticsearch.yml file. When I do this my masters spin up instead of crashing like before. But accessing /_cluster/health?pretty shows a red status:

{
  "cluster_name" : "elasticsearch-logs",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"
}

00 PM
Screen Shot 2017-10-30 at 12.11.00 PM.png1381×927 81.4 KB

Is there any other way available to run a cluster with 3 masters without modifying virtual memory? Thanks!

2

Each master is a node. I am not sure I understand how can you run 3 master nodes on single node cluster? Would it be at least a 3 node cluster? Perhaps you mean that 3 master nodes on a single machines? But then I am not quite sure what you are trying to achieve by doing that.

3

Sorry I had my terminology mixed up so I guess thats the reason the single node doesn't work. Yeah I want 3 master nodes on a single machine. I'm trying to combine the functionalities so that the 3 nodes are master/client/data. Here is the config for all 3:

elasticsearch.yml: |
    cluster.name: elasticsearch-logs
    node.name: ${HOSTNAME}
    node.master: true
    node.data: true
    network.host: _site_
    transport.tcp.port: 9300
    http.port: 9200
    http.enabled: true
    http.cors.enabled: true
    bootstrap.memory_lock: false
    xpack.security.enabled: false
    xpack.monitoring.enabled: false
    discovery.type: single-node
    discovery.zen.minimum_master_nodes: 2
    discovery.zen.ping_timeout: 10s
    discovery.zen.ping.unicast.hosts: elasticsearch-discovery

If it makes any difference, I'm trying to get the cluster running in a kubernetes cluster. So locally I am using a single node minikube cluster.

4

That is default. All nodes perform all these 3 functions by default.

But why do you want to run 3 nodes on the same machine?

5

Yeah that is what I'm trying to do

6

Why?

7

Locally I can only test/try to set up elasticsearch with minikube which is a single node cluster. So I can't split the nodes into different machines

I tried with only one elasticsearch node using the same config settings and still get the red status

[2017-10-30T16:24:24,883][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/47] [Main.cc@128] controller (64 bit): Version 5.6.3 (Build 5bf431d29b5bff) Copyright (c) 2017 Elasticsearch BV
[2017-10-30T16:24:24,928][INFO ][o.e.d.DiscoveryModule    ] [es-0] using discovery type [single-node]
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] initialized
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] starting ...
[2017-10-30T16:24:27,424][INFO ][o.e.t.TransportService   ] [es-0] publish_address {172.17.0.4:9300}, bound_addresses {172.17.0.4:9300}
[2017-10-30T16:24:27,495][WARN ][o.e.b.BootstrapChecks    ] [es-0] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-10-30T16:24:27,583][INFO ][o.e.c.s.ClusterService   ] [es-0] new_master {es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{172.17.0.4}{172.17.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: single-node-start-initial-join[{es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{172.17.0.4}{172.17.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}]
[2017-10-30T16:24:27,626][INFO ][o.e.h.n.Netty4HttpServerTransport] [es-0] publish_address {172.17.0.4:9200}, bound_addresses {172.17.0.4:9200}
[2017-10-30T16:24:27,626][INFO ][o.e.n.Node               ] [es-0] started

50 PM
Screen Shot 2017-10-30 at 12.42.50 PM.png1463×959 80.4 KB

8

I still don't understand why you want to run 3 nodes on the same machine, but if this is for testing, you can set transport interface to localhost and switch back to zen discovery.

9

Thanks! Would that be the transport.host: localhost config? Minikube is a vm so would I set the vm IP instead? Also what would change if I wanted to run just 1 node locally in my vm without modifying virtual memory.

EDIT: I was able to get it running without modifying memory by using a single node

10

Yes. You can disable bootstrap checks by running a single node cluster using single node discovery or by running several nodes using only localhost communication on the transport layer. In both cases, the cluster would be confined to a single machine.

I would also like to note that satisfying bootstraps checks is crucial for running elasticsearch in production. So, please check this blog post before trying to disable them.

1 Like
11

So in production if I was using 3 nodes in different machines, what could I do so that I don't have to modify the virtual memory?

12

In production you need to fix all bootstrap check issues.

1 Like
13

Alright, thanks!

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.