Nodes can't find each other in a single server setup

Elastic Stack Elasticsearch
,
1

Hi, I've been trying all day to run a 2-node cluster on a single server for testing purposes using Docker. But I keep getting:

"log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{hQ6MqkIaSDShNJXg9p4drQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9301] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see Troubleshooting discovery | Elasticsearch Guide [8.9] | Elastic"

These are the configurations, each on a different compose file:

Master:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - ./elasticsearch-certs:/usr/share/elasticsearch/config/certs
      - ./elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - node.name=node-1
      - network.publish_host=10.20.30.7
      - transport.publish_port:9300
      - cluster.initial_master_nodes=node-1
      - transport.host=localhost
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.enrollment.enabled=true
      - cluster.name=cluster

Non-master node:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    ports:
      - 9201:9200
      - 9301:9300
    volumes:
      - ./elasticsearch-certs:/usr/share/elasticsearch/config/certs
      - ./elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - node.roles=data
      - network.publish_host=10.20.30.7
      - transport.publish_port:9301
      - node.name=node-2
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - transport.host=localhost
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.enrollment.enabled=true
      - cluster.name=cluster
      - discovery.seed_hosts=10.20.30.7:9301

Has anyone successfully created a multi-node cluster using a different docker compose for each node? Or does anyone have any idea of what I might be doing wrong?

2

The docs linked from the log message say this:

To properly troubleshoot discovery and election problems, collect and analyse logs covering at least five minutes from all nodes.

Can you share such logs?

3

I don't see any relevant logs, but this is for the master branch:

elasticsearch-elasticsearch-1  | {"@timestamp":"2023-11-21T08:02:20.237Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-1][transport_worker][T#11]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"npZcYxdsS3GNeexWQg2zxg","elasticsearch.node.id":"tNszQqADQEmgABOgbLNZ4w","elasticsearch.node.name":"198_6_19_69-1","elasticsearch.cluster.name":"newsware"}
elasticsearch-elasticsearch-1  | {"@timestamp":"2023-11-21T08:02:20.349Z", "log.level": "INFO", "message":"Node [{198_6_19_69-1}{tNszQqADQEmgABOgbLNZ4w}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-1][management][T#1]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"npZcYxdsS3GNeexWQg2zxg","elasticsearch.node.id":"tNszQqADQEmgABOgbLNZ4w","elasticsearch.node.name":"198_6_19_69-1","elasticsearch.cluster.name":"newsware"}
elasticsearch-elasticsearch-1  | {"@timestamp":"2023-11-21T08:02:21.252Z", "log.level": "INFO",  "current.health":"YELLOW","message":"Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.apm-source-map][0]]]).","previous.health":"RED","reason":"shards started [[.apm-source-map][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"npZcYxdsS3GNeexWQg2zxg","elasticsearch.node.id":"tNszQqADQEmgABOgbLNZ4w","elasticsearch.node.name":"198_6_19_69-1","elasticsearch.cluster.name":"newsware"}
elasticsearch-elasticsearch-1  | {"@timestamp":"2023-11-21T08:13:28.210Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.18.0.3:9200, remoteAddress=/172.18.0.1:53908}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-1][transport_worker][T#31]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"npZcYxdsS3GNeexWQg2zxg","elasticsearch.node.id":"tNszQqADQEmgABOgbLNZ4w","elasticsearch.node.name":"198_6_19_69-1","elasticsearch.cluster.name":"newsware"}
elasticsearch-elasticsearch-1  | {"@timestamp":"2023-11-21T08:15:28.355Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"kibana","elasticsearch.event.category":"api","elasticsearch.http.request.x_opaque_id":"unknownId;kibana:task%20manager:run%20security%3Atelemetry-prebuilt-rule-alerts:security%3Atelemetry-prebuilt-rule-alerts%3A1.0.0","event.code":"deprecated_field_exclude","message":"Deprecated field [exclude] used, expected [excludes] instead" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[198_6_19_69-1][transport_worker][T#1]","log.logger":"org.elasticsearch.deprecation.xcontent.ParseField","trace.id":"345eb9717306524040bbc919a02641d9","elasticsearch.cluster.uuid":"npZcYxdsS3GNeexWQg2zxg","elasticsearch.node.id":"tNszQqADQEmgABOgbLNZ4w","elasticsearch.node.name":"198_6_19_69-1","elasticsearch.cluster.name":"newsware"}

And for the non-master branch, the same message is repeated over and over:

estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:24.021Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:25.021Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:26.022Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:27.023Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:28.023Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:29.024Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:30.023Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:21:31.022Z", "log.level": "WARN", "message":"address [10.20.30.7:9301], node [null], requesting [false] discovery result: [][10.20.30.7:9301] connect_exception: Failed execution: java.nio.channels.ClosedChannelException: null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}

This message also shows repeatedly, although at a much slower rate:

estest-elasticsearch-1  | {"@timestamp":"2023-11-21T09:24:43.103Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{198_6_19_69-2}{k8wUeUVARnSsj_qXfhW3oQ}{hQ6MqkIaSDShNJXg9p4drQ}{198_6_19_69-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9301] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[198_6_19_69-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"198_6_19_69-2","elasticsearch.cluster.name":"newsware"}

As an extra note, I'm able to curl one another from inside the docker containers.

4

Should the port mapping not match what the node is listening to, e.g. mapping 9301:9301?

5

That's not the full logs so I'm still guessing a little, but the excerpt you shared suggests that the master node's transport port is not accessible at 10.20.30.7:9301.

6

I set up a new testing environment, here are the configuration and full logs since starting the node. I'm able to curl the other node from inside each container, I get a message like:

sh-5.0$ curl 10.20.30.7:9302
curl: (56) Recv failure: Connection reset by peer

Master config (running on port 9302):

    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    ports:
      - 9202:9200
      - 9302:9300
    restart: always
    volumes:
      - ./elasticsearch-certs:/usr/share/elasticsearch/config/certs
      - ./elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - node.name=node-1
      - cluster.name=cluster
      - cluster.initial_master_nodes=node-1
      - network.publish_host=10.20.30.7
      - transport.publish_port:9302
      - transport.host=localhost
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.enrollment.enabled=true

Non-master config (running on port 9301):

    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    ports:
      - 9201:9200
      - 9301:9300
    restart: always
    volumes:
      - ./elasticsearch-certs:/usr/share/elasticsearch/config/certs
      - ./elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - node.name=node-2
      - cluster.name=cluster
      - discovery.seed_hosts=10.20.30.7:9302
      - network.publish_host=10.20.30.7
      - transport.publish_port:9301
      - transport.host=localhost
      - node.roles=data
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.enrollment.enabled=true
7

Master logs:

...
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.960Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.961Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.961Z", "log.level": "INFO", "message":"loaded module [ml-package-loader]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.961Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.961Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.962Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.962Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.962Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.963Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.963Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.963Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.963Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.963Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.964Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.965Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.965Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.965Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.965Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.965Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [blob-cache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.966Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.967Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.967Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.967Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.967Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [x-pack-profiling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [x-pack-write-load-forecaster]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.968Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.969Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.969Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.969Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.969Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.969Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.970Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:04.970Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:10.363Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda3)]], net usable_space [1018.3gb], net total_space [1tb], types [ext4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:10.364Z", "log.level": "INFO", "message":"heap size [31gb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:10.496Z", "log.level": "INFO", "message":"node name [node-1], node ID [LfVxFcThRsyyPo5qprTmAA], cluster name [cluster], roles [master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:14.909Z", "log.level": "INFO", "message":"[controller/260] [Main.cc@123] controller (64 bit): Version 8.9.0 (Build 4b769f85c816b8) Copyright (c) 2023 Elasticsearch BV", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:15.119Z", "log.level": "INFO", "message":"Security is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:15.988Z", "log.level": "INFO", "message":"parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.authz.store.FileRolesStore","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:16.511Z", "log.level": "INFO", "message":"Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.InitialNodeSecurityAutoConfiguration","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:17.178Z", "log.level": "INFO", "message":"Profiling is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:17.200Z", "log.level": "INFO", "message":"profiling index templates will not be installed or reinstalled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:18.350Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:18.388Z", "log.level": "INFO", "message":"using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:18.447Z", "log.level": "INFO", "message":"using discovery type [multi-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.133Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.134Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.150Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.151Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.365Z", "log.level": "INFO", "message":"publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.831Z", "log.level": "WARN", "message":"memory locking requested for elasticsearch process but memory is not locked", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.831Z", "log.level": "WARN", "message":"max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.833Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [ZOrf4bzfQ2q7QG0ZP8_y7Q] but [cluster.initial_master_nodes] is set to [node-1]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:20.959Z", "log.level": "INFO", "message":"elected-as-master ([1] nodes joined in term 2)[_FINISH_ELECTION_, {node-1}{LfVxFcThRsyyPo5qprTmAA}{kjZOJmUhS1CfG522c-iHJg}{node-1}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{8.9.0} completing election], term: 2, version: 37, delta: master node changed {previous [], current [{node-1}{LfVxFcThRsyyPo5qprTmAA}{kjZOJmUhS1CfG522c-iHJg}{node-1}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{8.9.0}]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.010Z", "log.level": "INFO", "message":"master node changed {previous [], current [{node-1}{LfVxFcThRsyyPo5qprTmAA}{kjZOJmUhS1CfG522c-iHJg}{node-1}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{8.9.0}]}, term: 2, version: 37, reason: Publication{term=2, version=37}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.052Z", "log.level": "INFO", "message":"starting file watcher ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.file.AbstractFileWatchingService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.140Z", "log.level": "INFO", "message":"file settings service up and running [tid=98]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[file-watcher[/usr/share/elasticsearch/config/operator/settings.json]]","log.logger":"org.elasticsearch.common.file.AbstractFileWatchingService","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.147Z", "log.level": "INFO", "message":"publish_address {10.20.30.7:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.147Z", "log.level": "INFO", "message":"node-join: [{node-1}{LfVxFcThRsyyPo5qprTmAA}{kjZOJmUhS1CfG522c-iHJg}{node-1}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{8.9.0}] with reason [completing election]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.coordination.NodeJoinExecutor","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.147Z", "log.level": "INFO", "message":"started {node-1}{LfVxFcThRsyyPo5qprTmAA}{kjZOJmUhS1CfG522c-iHJg}{node-1}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{8.9.0}{ml.machine_memory=101294649344, ml.allocated_processors=32, ml.max_jvm_size=33285996544, ml.allocated_processors_double=32.0, xpack.installed=true}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.628Z", "log.level": "INFO", "message":"license [405b774d-d006-4a85-9009-0b17fccf7054] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.630Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.634Z", "log.level": "INFO", "message":"recovered [1] indices into cluster_state", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.gateway.GatewayService","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:21.720Z", "log.level": "INFO", "message":"Node [{node-1}{LfVxFcThRsyyPo5qprTmAA}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][management][T#2]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
estest2-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:22.175Z", "log.level": "INFO",  "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0]]]).","previous.health":"RED","reason":"shards started [[.security-7][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-1][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"ZOrf4bzfQ2q7QG0ZP8_y7Q","elasticsearch.node.id":"LfVxFcThRsyyPo5qprTmAA","elasticsearch.node.name":"node-1","elasticsearch.cluster.name":"cluster"}
8

Non-master logs:

...
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.575Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.575Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.575Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.575Z", "log.level": "INFO", "message":"loaded module [ml-package-loader]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.575Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.576Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.576Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.576Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.576Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.577Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.577Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.577Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.577Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.577Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.578Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.579Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.579Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.579Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.579Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.579Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [blob-cache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.580Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.581Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.581Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.581Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [x-pack-profiling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [x-pack-write-load-forecaster]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.582Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.583Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.584Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.584Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:35.584Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:39.669Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda3)]], net usable_space [1018.3gb], net total_space [1tb], types [ext4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:39.669Z", "log.level": "INFO", "message":"heap size [31gb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:39.733Z", "log.level": "INFO", "message":"node name [node-2], node ID [k8wUeUVARnSsj_qXfhW3oQ], cluster name [cluster], roles [data]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:42.796Z", "log.level": "INFO", "message":"[controller/251] [Main.cc@123] controller (64 bit): Version 8.9.0 (Build 4b769f85c816b8) Copyright (c) 2023 Elasticsearch BV", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:42.992Z", "log.level": "INFO", "message":"Security is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:43.631Z", "log.level": "INFO", "message":"parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.authz.store.FileRolesStore","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:44.133Z", "log.level": "INFO", "message":"Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.InitialNodeSecurityAutoConfiguration","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:44.434Z", "log.level": "INFO", "message":"Profiling is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:44.449Z", "log.level": "INFO", "message":"profiling index templates will not be installed or reinstalled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:45.124Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:45.149Z", "log.level": "INFO", "message":"using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:45.193Z", "log.level": "INFO", "message":"using discovery type [multi-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:46.806Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:46.807Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:46.821Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:46.822Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:46.924Z", "log.level": "INFO", "message":"publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:47.098Z", "log.level": "WARN", "message":"memory locking requested for elasticsearch process but memory is not locked", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:47.098Z", "log.level": "WARN", "message":"max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:47.100Z", "log.level": "INFO", "message":"this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:46:57.116Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:07.117Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:17.117Z", "log.level": "WARN", "message":"timed out while waiting for initial discovery state - timeout: 30s", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][generic][T#3]","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:17.119Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:17.125Z", "log.level": "INFO", "message":"publish_address {10.20.30.7:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:17.126Z", "log.level": "INFO", "message":"started {node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}{xpack.installed=true}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:27.121Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:37.122Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:47.124Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}
estest-elasticsearch-1  | {"@timestamp":"2023-11-21T18:47:57.125Z", "log.level": "WARN", "message":"master not discovered yet: have discovered [{node-2}{k8wUeUVARnSsj_qXfhW3oQ}{8CFV2CyZQymA4JmBYnhMDQ}{node-2}{localhost}{127.0.0.1:9300}{d}{8.9.0}]; discovery will continue using [10.20.30.7:9302] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-2][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"node-2","elasticsearch.cluster.name":"cluster"}

Thanks for the help, I think my configuration is valid, can't find where things are going wrong.

9

Did you try changing the port mapping so you map 9301 to 9301, which is what Elasticsearch is listening on, on node-2? That way it would look like node-1, which seems correct to me.

10

Yep that's consistent with my earlier guess, the node is not accessible at this address.

11

Thanks @DavidTurner and @Christian_Dahlqvist

The issue was that I had transport.host set to localhost, so it was listening only in the container locahost network. I fixed it by setting it to 0.0.0.0, setting network.host to 0.0.0.0 works too.

Setting it to 0.0.0.0 made bootstrapping fail and the nodes wouldn't start, I had to solve the errors from the log including setting a bigger value for vm.max_map_count and setting ulimits.memlock to -1 for both soft and hard in the docker compose.

Hope this helps someone with the same issue.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.