RHEL6 ICMP ipv6 error at startup

Elastic Stack Beats
1

Hi,

I'm running into issues when trying to do an ICMP ping with heartbeat to a different server. I'm getting some error messages on startup:

2017-05-03T10:27:26+02:00 INFO Select (active) monitor icmp
2017-05-03T10:27:26+02:00 INFO IPv6 ICMP not supported: listen ip6:ipv6-icmp : socket: address family not supported by protocol
2017-05-03T10:27:26+02:00 ERR failed to load monitor tasks: failed to initiate IPv6 support when initializing monitor icmp(0)
2017-05-03T10:27:26+02:00 INFO heartbeat start running.
2017-05-03T10:27:26+02:00 INFO heartbeat is running! Hit CTRL-C to stop it.
2017-05-03T10:27:56+02:00 INFO No non-zero metrics in the last 30s

At first I got the same message for IPv4 as well, but after some googling I noticed that this was an issue noticed by a beats-docker user here: https://github.com/elastic/beats-docker/issues/2

The work around is running a command that's already incorperated in the packetbeat binary: setcap cap_net_raw,cap_net_admin=eip /<path/to/>heartbeat

This allows the user which is running beats to open raw sockets (if I understood correctly). After issuing this command the error messages concerning ipv4 were gone. The monitor however still fails completely and won't run because of the ipv6 ping that can't be done.

So I can think of 2 solutions to my problem:

  1. find out if there's a way to grant rights to heartbeat to allow ipv6 ping/sockets similar to ipv4.
  2. disable ipv6 ping and force heartbeat to only use ipv4 ping.

Does anyone know if this is possible and how? If there are different solutions to my problem I'm all ears too.

Thanks!

edit:

I'm using heartbeat-5.3.0-linux-x86_64 on RHEL6.

config for the icmp monitor is:

  • type: icmp
    schedule: '@every 1m'
    hosts: ["10.10.10.10"]

(manually pinging the ip adress works fine under the user that runs heartbeat)

2

I managed to get it working with setting putting the following setting on the heartbeat executable: setcap cap_net_raw,cap_net_admin=eip heartbeat

I also disabled ipv6 in the heartbeat.yml, which turns the error into a warning in the logging.

Be warned though, if you change ownership (like I did for testing between the user I created to run beats and root), you need to reset this command.

3

Yeah, packetbeat and ICMP ping do require root rights. Users seem to use HTTP mode most the time. We might add UDP based ping as well, as is often used with ping when run by non-root users.

4

My last post actually provided a workaround so you can run it under a different user than root. This workaround/fix is already implement for beats-docker, it would be nice if you could also implement it for heartbeat.

The fix for beats-docker can be found here: https://github.com/elastic/beats-docker/issues/2

Should I create an issue on github for heartbeat for this?

5

To me it looks like a docker problem + sounds like it's already resolved.

Disabling ipv6 depends on the kernel I think. e.g. If IPv6 capabilities are disabled on boot, you must disable it in heartbeat as well.

Should I create an issue on github for heartbeat for this?

Sorry, I am a little confused here. What's the difference between your issue and the one you linked? If you think there is a difference a bug report is very welcome.

6

Looking back I see it might be a bit confusing. The ipv6 issue was indeed sovled by disabling ipv6 in heartbeat.yml.

There is however another issue. If you try to run heartbeat as a non root user, which is pretty common sense considering all things you run on linux, you run into a different error:

2017/02/24 02:53:36.459997 loop.go:330: INFO IPv4 ICMP not supported: listen ip4:icmp : socket: operation not permitted
2017/02/24 02:53:36.460013 loop.go:330: INFO IPv6 ICMP not supported: listen ip6:ipv6-icmp : socket: operation not permitted

That error is the same one as the error for beats-docker. It's not docker related but a permissions issue. Basically non-root users cannot open a raw tcp socket, which is required to do an icmp ping. I guess you guys implemented fast-ping, instead of the default /bin/ping on linux distros, since fast-ping seems to have the same issues while the default /bin/ping can be used by any user under linux normally.

For beats-docker this issue seems to be also affecting the root user so there they had to fix it because otherwise it wouldn't work at all. What they did it run the "setcap cap_net_raw,cap_net_admin=eip heartbeat" comand (where heartbeat is the heartbeat executable). If you change ownership of the heartbeat files to the the user you want to run heartbeat as (a non-root account) and then use the setcap command for the heartbeat file, it will allow you to run heartbeat with ping monitors as a non-root user.

The documentation now states that ping can only be used when running beats as root (which is a really bad practise..), but this solution allows you to run it as other users as well. Is there a formal process to get this checked out and maybe added to the documentation?

7

I like the approach of setting capabilities, instead of requiring root all the time.

For docs, you can simply open a github issue. Maybe mention where you would add this and what content you would expect. We will pick up and label the issue accordingly.

If you are adventurous you can open a PR with documentation changes. The heartbeat docs are in the docs folder.

Well, for ICMP you need to open a raw socket. That is you need to have the credentials to do so. Check your OS, ping might have the suid flag or the cat_net_raw capability be set. Some tools also fallback to UDP either trying an unlikely port number or a known port number with known protocol (e.g. DNS). I think traceroute supports this by setting packet TTL via setsockopt .

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.