Error while trying to new node to existing cluster. Bootstrap settings failed

[charvi23 (charvi)]

Getting error while adding new node to cluster.

[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch. For more information see [
https://www.elastic.co/guide/en/elasticsearch/reference/8.12/bootstrap-checks.html]
bootstrap check failure [1] of [1]: system call filters failed to install; check the logs and fix your configuration; for more information see [
https://www.elastic.co/guide/en/elasticsearch/reference/8.12/_system_call_filter_check.html]
[2024-02-02T13:31:05,619][WARN ][o.e.n.Node               ] [node01] unexpected exception while waiting for http server to close
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't move to stopped state when not started
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
        at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
        at org.elasticsearch.node.Node.prepareForClose(Node.java:594) ~[elasticsearch-8.12.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.shutdown(Elasticsearch.java:468) ~[elasticsearch-8.12.0.jar:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.IllegalStateException: Can't move to stopped state when not started
        at org.elasticsearch.common.component.Lifecycle.canMoveToStopped(Lifecycle.java:128) ~[elasticsearch-8.12.0.jar:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:73) ~[elasticsearch-8.12.0.jar:?]
        at org.elasticsearch.node.Node.lambda$prepareForClose$25(Node.java:584) ~[elasticsearch-8.12.0.jar:?]

Gone through the bootstrap document, not able to figure out error that might be causing it.

Elastic search version on master: 8.8
ES version on new node: 8.12.

elasticsearch.yml (New node)

discovery.seed_hosts: ["x.x.x.x:9200", "x.x.x.x:9200"]
 Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate 
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

[charvi23 (charvi)]

I was able to figure out the logs and issue. If it might help someone adding it here:

Initial logs also showed following error:

unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-x86/libjnidispatch.so

Resolution:

Set ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" in /etc/sysconfig/elasticsearch
mkdir -p /var/lib/elasticsearch/tmp
chown -R elasticsearch.elasticsearch /var/lib/elasticsearch/tmp

Referred from linux - Unable to load JNA native support library Elasticsearch 6.x - Stack Overflow

[DavidTurner]

It's best to set ES_TMPDIR instead of -Djna.tmpdir; see these docs for more details.