Bypassing failing bootstrap checks

I am trying to run 3 elasticsearch masters on a single node cluster but am having issues regarding the virtual memory setting. Everything seems to be running fine locally if I create the cluster/masters after first using sysctl to increase the virtual memory but the issue is that in production we do not run in privileged mode/allow modification to the virtual memory so I cannot add the sysctl procedure to create my elasticsearch cluster. I tried to get the cluster to start up and run locally without the sysctl beforehand and got the following message:

[2017-10-30T15:46:12,338][INFO ][o.e.b.BootstrapChecks    ] [es-0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Although it isn't recommended, I also tried by disabling the bootstrap checks by adding the discovery.type: single-node config to my elasticsearch.yml file. When I do this my masters spin up instead of crashing like before. But accessing /_cluster/health?pretty shows a red status:

  "cluster_name" : "elasticsearch-logs",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"

Is there any other way available to run a cluster with 3 masters without modifying virtual memory? Thanks!

Each master is a node. I am not sure I understand how can you run 3 master nodes on single node cluster? Would it be at least a 3 node cluster? Perhaps you mean that 3 master nodes on a single machines? But then I am not quite sure what you are trying to achieve by doing that.

Sorry I had my terminology mixed up so I guess thats the reason the single node doesn't work. Yeah I want 3 master nodes on a single machine. I'm trying to combine the functionalities so that the 3 nodes are master/client/data. Here is the config for all 3:

elasticsearch.yml: | elasticsearch-logs ${HOSTNAME}
    node.master: true true _site_
    transport.tcp.port: 9300
    http.port: 9200
    http.enabled: true
    http.cors.enabled: true
    bootstrap.memory_lock: false false
    xpack.monitoring.enabled: false
    discovery.type: single-node
    discovery.zen.minimum_master_nodes: 2
    discovery.zen.ping_timeout: 10s elasticsearch-discovery

If it makes any difference, I'm trying to get the cluster running in a kubernetes cluster. So locally I am using a single node minikube cluster.

That is default. All nodes perform all these 3 functions by default.

But why do you want to run 3 nodes on the same machine?

Yeah that is what I'm trying to do


Locally I can only test/try to set up elasticsearch with minikube which is a single node cluster. So I can't split the nodes into different machines

I tried with only one elasticsearch node using the same config settings and still get the red status

[2017-10-30T16:24:24,883][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/47] [] controller (64 bit): Version 5.6.3 (Build 5bf431d29b5bff) Copyright (c) 2017 Elasticsearch BV
[2017-10-30T16:24:24,928][INFO ][o.e.d.DiscoveryModule    ] [es-0] using discovery type [single-node]
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] initialized
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] starting ...
[2017-10-30T16:24:27,424][INFO ][o.e.t.TransportService   ] [es-0] publish_address {}, bound_addresses {}
[2017-10-30T16:24:27,495][WARN ][o.e.b.BootstrapChecks    ] [es-0] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-10-30T16:24:27,583][INFO ][o.e.c.s.ClusterService   ] [es-0] new_master {es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{}{}{ml.max_open_jobs=10, ml.enabled=true}, reason: single-node-start-initial-join[{es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{}{}{ml.max_open_jobs=10, ml.enabled=true}]
[2017-10-30T16:24:27,626][INFO ][o.e.h.n.Netty4HttpServerTransport] [es-0] publish_address {}, bound_addresses {}
[2017-10-30T16:24:27,626][INFO ][o.e.n.Node               ] [es-0] started

I still don't understand why you want to run 3 nodes on the same machine, but if this is for testing, you can set transport interface to localhost and switch back to zen discovery.

Thanks! Would that be the localhost config? Minikube is a vm so would I set the vm IP instead? Also what would change if I wanted to run just 1 node locally in my vm without modifying virtual memory.

EDIT: I was able to get it running without modifying memory by using a single node

Yes. You can disable bootstrap checks by running a single node cluster using single node discovery or by running several nodes using only localhost communication on the transport layer. In both cases, the cluster would be confined to a single machine.

I would also like to note that satisfying bootstraps checks is crucial for running Elasticsearch in production. So, please check this blog post before trying to disable them.

1 Like

So in production if I was using 3 nodes in different machines, what could I do so that I don't have to modify the virtual memory?

In production you need to fix all bootstrap check issues.

1 Like

Alright, thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.