Bootstrap Checks trouble on non-root access cluster(CentOS)

[mastayoda]

Continuing discussion in github issue 21655

Brief:

We are building a graph database in top of ElasticSearch( trueno). Previously we where using ES 2.3 without problem to form a cluster of 20 machines using CentOS:

	Icon name: computer-server
	Chassis: server
	Machine ID: 4624fe6856e646b8ac7b3f737f691f2a
	Boot ID: 9f37aa289d484c2d9d98fa64319291c5
	Operating System: CentOS Linux 7 (Core)
	CPE OS Name: cpe:/o:centos:centos:7
	Kernel: Linux 3.10.0-327.el7.x86_64
	Architecture: x86-64

We don't have root access to these 20 machines, but since ES 2.3 did not had Bootstrap checks we where able to setup a 20 node cluster easily. Now we are migrating to ES 5.3 to take advantage of the Sliced Scroll feature and read data in parallel in order to speedup graph computation in Spark, but we cannot setup the cluster because we need the 20 machines to communicate between themselves, and ES automatically falls into production mode which enforces Bootstrap checks. Some of them can be solved without root and others strictly needs root(such as file descriptors). Ours scenario is a bit uncommon and we agree Bootstrap checks are a good idea since it prevent disasters, but in our case it is not necessary since Trueno will be used as a storage engine mainly for experiments and analytics in research environments. It may be a good idea to provide a setting to bypass these checks.

Thanks in advanced!

[jasontedor]

How do you start Elasticsearch? Are you using systemd or are you starting from bin/elasticsearch directly?

[mastayoda]

At this moment from bin/elasticsearch

We overwrite the configuration with the following settings:

    cluster.name: "trueno"
    node.name: ${HOSTNAME}
    http.port: 8004
    network.host: 0.0.0.0
    transport.tcp.port: 9300
    transport.host: localhost
    action.auto_create_index: false
    discovery.zen.ping.unicast.hosts: ["localhost"]
    path.data: "elasticsearch"
    indices.requests.cache.size: "35%"
    indices.queries.cache.size: "30%"
    bootstrap.system_call_filter: false

All the 'localhost' are configurable to the user. For example, if the user starts our database as:

    trueno start -h myhostname

it will execute bin/elasticsearch, and

      - 'transport.host'
      - 'network.host'
      - 'discovery.zen.ping.unicast.hosts'

will be overwritten to 'myhostname'

[jasontedor]

How were you able to get past the max map count check? At least, that's what I understand from your message:

I may increase the memory, but the file descriptor needs root access.

[mastayoda]

I mean that some of the checks can be solved without root(like increasing the Java Virtual Machine Heap before running the process). But those like the file descriptors needs root. I have tried workarounds for increasing file descriptors without root access, but no success so far. I have also tried to use several flags to bypass the checks but are not present anymore in 5.3, ES throws an error of unrecognized option.

[jasontedor]

Okay, I'm sorry to tell you that you need to get your system administrator to configure these values for you.

Alternatively, you can have the system administrator install the RPM package which sets the max map count on package install, and sets the number of file descriptors for the service.

[mastayoda]

Jason, no worries, we will contact our system administrator for help. Thanks so much for taking your time to help, I deeply appreciate it! Also wonderful job with ElasticSearch, you guys are heroes.

[jasontedor]

I appreciate your kind words here.

I know the bootstrap checks can be a pain but we think they are very important to help users detect misconfiguration that would lead to data loss.

[mastayoda]

100% agree!