Bypassing failing bootstrap checks

Elastic Stack Elasticsearch
1

I am trying to run 3 elasticsearch masters on a single node cluster but am having issues regarding the virtual memory setting. Everything seems to be running fine locally if I create the cluster/masters after first using sysctl to increase the virtual memory but the issue is that in production we do not run in privileged mode/allow modification to the virtual memory so I cannot add the sysctl procedure to create my elasticsearch cluster. I tried to get the cluster to start up and run locally without the sysctl beforehand and got the following message:

[2017-10-30T15:46:12,338][INFO ][o.e.b.BootstrapChecks    ] [es-0] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Although it isn't recommended, I also tried by disabling the bootstrap checks by adding the discovery.type: single-node config to my elasticsearch.yml file. When I do this my masters spin up instead of crashing like before. But accessing /_cluster/health?pretty shows a red status:

{
  "cluster_name" : "elasticsearch-logs",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : "NaN"
}

00 PM
Screen Shot 2017-10-30 at 12.11.00 PM.png1381×927 81.4 KB

Is there any other way available to run a cluster with 3 masters without modifying virtual memory? Thanks!

2

Each master is a node. I am not sure I understand how can you run 3 master nodes on single node cluster? Would it be at least a 3 node cluster? Perhaps you mean that 3 master nodes on a single machines? But then I am not quite sure what you are trying to achieve by doing that.

3

Sorry I had my terminology mixed up so I guess thats the reason the single node doesn't work. Yeah I want 3 master nodes on a single machine. I'm trying to combine the functionalities so that the 3 nodes are master/client/data. Here is the config for all 3:

elasticsearch.yml: |
    cluster.name: elasticsearch-logs
    node.name: ${HOSTNAME}
    node.master: true
    node.data: true
    network.host: _site_
    transport.tcp.port: 9300
    http.port: 9200
    http.enabled: true
    http.cors.enabled: true
    bootstrap.memory_lock: false
    xpack.security.enabled: false
    xpack.monitoring.enabled: false
    discovery.type: single-node
    discovery.zen.minimum_master_nodes: 2
    discovery.zen.ping_timeout: 10s
    discovery.zen.ping.unicast.hosts: elasticsearch-discovery

If it makes any difference, I'm trying to get the cluster running in a kubernetes cluster. So locally I am using a single node minikube cluster.

4

That is default. All nodes perform all these 3 functions by default.

But why do you want to run 3 nodes on the same machine?

5

Yeah that is what I'm trying to do

6

Why?

7

Locally I can only test/try to set up elasticsearch with minikube which is a single node cluster. So I can't split the nodes into different machines

I tried with only one elasticsearch node using the same config settings and still get the red status

[2017-10-30T16:24:24,883][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/47] [Main.cc@128] controller (64 bit): Version 5.6.3 (Build 5bf431d29b5bff) Copyright (c) 2017 Elasticsearch BV
[2017-10-30T16:24:24,928][INFO ][o.e.d.DiscoveryModule    ] [es-0] using discovery type [single-node]
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] initialized
[2017-10-30T16:24:26,904][INFO ][o.e.n.Node               ] [es-0] starting ...
[2017-10-30T16:24:27,424][INFO ][o.e.t.TransportService   ] [es-0] publish_address {172.17.0.4:9300}, bound_addresses {172.17.0.4:9300}
[2017-10-30T16:24:27,495][WARN ][o.e.b.BootstrapChecks    ] [es-0] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-10-30T16:24:27,583][INFO ][o.e.c.s.ClusterService   ] [es-0] new_master {es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{172.17.0.4}{172.17.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: single-node-start-initial-join[{es-0}{aoQp8O2QRMOH_sjS8iBstg}{oL2QdhLdTgmhdGedpTbN7w}{172.17.0.4}{172.17.0.4:9300}{ml.max_open_jobs=10, ml.enabled=true}]
[2017-10-30T16:24:27,626][INFO ][o.e.h.n.Netty4HttpServerTransport] [es-0] publish_address {172.17.0.4:9200}, bound_addresses {172.17.0.4:9200}
[2017-10-30T16:24:27,626][INFO ][o.e.n.Node               ] [es-0] started

50 PM
Screen Shot 2017-10-30 at 12.42.50 PM.png1463×959 80.4 KB

8

I still don't understand why you want to run 3 nodes on the same machine, but if this is for testing, you can set transport interface to localhost and switch back to zen discovery.

9

Thanks! Would that be the transport.host: localhost config? Minikube is a vm so would I set the vm IP instead? Also what would change if I wanted to run just 1 node locally in my vm without modifying virtual memory.

EDIT: I was able to get it running without modifying memory by using a single node

10

Yes. You can disable bootstrap checks by running a single node cluster using single node discovery or by running several nodes using only localhost communication on the transport layer. In both cases, the cluster would be confined to a single machine.

I would also like to note that satisfying bootstraps checks is crucial for running elasticsearch in production. So, please check this blog post before trying to disable them.

11

So in production if I was using 3 nodes in different machines, what could I do so that I don't have to modify the virtual memory?

12

In production you need to fix all bootstrap check issues.

13

Alright, thanks!