Elasticsearch will not properly boot without localhost settings

Elastic Stack Elasticsearch
1

Hello everyone.

I've exhausted nearly all forms of documentation and forum searches, and I've yet been able to get elasticsearch to outside of local settings.

For some background: I need elasticsearch to be able to communicate with Grafana, which will need to carry over the logs for the Wazuh App for Kibana. I'm using a distributed setup with Wazuh, so the wazuh manager is its own server apart from the rest.

Regardless of the yml/java bootstrap settings I put place, and regardless of my assigning the ELK instance's public IP to 'network.host' and 'cluster.initial_master_nodes', it can't bind to this address. Using the private IP is out of the question, especially if it's to be able to do talk with the grafana server on a different subnet. See the errors below:

[2020-06-25T20:00:32,609][INFO ][o.e.p.PluginsService     ] [wazuh-log-vizualizer] loaded module [x-pack-watcher]
[2020-06-25T20:00:32,609][INFO ][o.e.p.PluginsService     ] [wazuh-log-vizualizer] no plugins loaded
[2020-06-25T20:00:32,675][INFO ][o.e.e.NodeEnvironment    ] [wazuh-log-vizualizer] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [1.5gb], net total_space [7.6gb], types [ext4]
 [2020-06-25T20:00:32,676][INFO ][o.e.e.NodeEnvironment    ] [wazuh-log-vizualizer] heap size [4gb], compressed ordinary object pointers [true]
[2020-06-25T20:00:32,882][INFO ][o.e.n.Node               ] [wazuh-log-vizualizer] node name [wazuh-log-vizualizer], node ID [MlirIa3rSQWsPxo5rReRSw], cluster name [elasticsearch]
[2020-06-25T20:00:36,455][ERROR][o.e.b.Bootstrap          ] [wazuh-log-vizualizer] Exception
java.lang.IllegalArgumentException: unknown setting [cluster.initial master nodes] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

AND

[2020-06-26T15:13:00,968][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [wazuh-log-vizualizer] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to 3.x.x.x:[9300-9400]]; nested: BindException[Cannot assign requested address];
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
            at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
    Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to 3.x.x.x:[9300-9400]

If anyone can give any insight, i'd be hugely grateful.

2

Welcome!

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Here I think this message says it all:

[2020-06-25T20:00:36,455][ERROR][o.e.b.Bootstrap ] [wazuh-log-vizualizer] Exception
java.lang.IllegalArgumentException: unknown setting [cluster.initial master nodes] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

cluster.initial master nodes is not a know setting. Did you set cluster.initial_master_nodes or cluster.initial master nodes?

Could you share your full and correctly formatted elasticsearch.yml file?

3

The configuration settings contained in my yml file are below:

node.name: "wazuh-log-vizualizer"

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: "3.x.x.x"

cluster.initial_master_nodes: ["3.x.x.x"]

As for cluster.initial master nodes, neither formattings work for me.

EDIT: The above, meaning, that cluster.initial_master_nodes with underscores appear to work, where the spaces do not. Regardless, the IP can't be bound.

4

Bump.

Any help would be huge in this, and I'm getting to a point where getting blocked is no longer an option. Thank you.

5

Is this IP available on a network card on this machine?

6

Yes it is, and that's why this was such a big mystery/headache for me; I didn't read any documentation about public IPs being unusable in this way.

Since my asking this question however, I gave up on using the public IP, and went with the private IP instead. It's now booting like it should be without using the loopback address. This is do-able, because both my elasticstack server and my (Wazuh) application server are in the same VPC.

My current issue now is in getting Wazuh logs on my Wazuh manager over to the elasticstack server, using filebeat, for browsing over the Wazuh Kibana app. I've sought assistance from Wazuh support on this part, as I don't believe the issue lies with filebeat or elasticsearch at this point. I can sift through just about every other feature/metric except for the logs themselves.

7

I was able to get the issue above (my ultimate goal), sorted out. Thanks for responding to my questions - I appreciate it.

If there are any insights you may have on why a public IP address for elasticsearch wouldn't bind in this way (an EIP issued by AWS), I'd be happy to hear it.

8

Is the public IP bound to a network card? Or is it more like a proxy in the middle?

Can you share the network card details here? Like "ifconfig" or similar.

9

The stack server itself is reachable only through a bastion proxy. Kibana utilizes this IP for access to its web ui.

Ifconfig doesn't list the public IP in its details, however, the address is bound to the network card device. This is an aws instance in a VPC on a public subnet, so I don't think there should be any reason why it isn't possible to use it.

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.x.x.x  netmask 255.255.255.0  broadcast 10.x.x.255
        inet6 fe80::826:41ff:feb6:dbd7  prefixlen 64  scopeid 0x20<link>
        ether 0a:26:41:b6:db:d7  txqueuelen 1000  (Ethernet)
        RX packets 375927  bytes 457181416 (457.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 129221  bytes 72724273 (72.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
10

I "think" (but unsure as not a network/ops guy) that you can only bind to 10.x.x.x in your case. But you can "publish" the public IP 3.x.x.x to the rest of the cluster using network.publish_host. See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html.

Note (not directly related): I also remember that with the ec2 discovery plugin, you can also make this IP address "dynamic" with https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html#discovery-ec2-network-host.

11

Thank you very much. I'll look into this.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.