Heartbeat sighup received

[ea1987 (andrea)]

Hi all,
I usually start Heartbeat (and other beats) with the following command:

nohup ./beatname >/dev/null 2>&1 &

but heartbeat stops after a while (random period). I checked on debug logs and this is the message I found:

2019-09-26T13:00:36.933+0200 DEBUG [service] service/service.go:55 Received sighup, stopping

Of course, sighup is related to an iterrupt of the process related to the user disconnection from the terminal, but this is strange cause sometimes heartbeat runs for hour after user disconnection.
NB: Elasticsearch and Kibana are running in the same server and are started using the same command, but they are not stopped.

According to your experience what could be the cause of this issue?
Thanks

[Andrew_Cholakian1 (Andrew Cholakian)]

This is the first such report we've seen unfortunately. Is there some other thing on the server that could be sending SIGHUPs?

SIGHUP can be sent via kill or from systemd and similar. It's very hard to say without investigating one's full environment.

[ea1987 (andrea)]

Well, I have performed a brief check on all main ELK components about their signal managements and I noticed that beats, apm and node (Kibana) handle sighup command by theirself, as can be seen looking at the last SigIn digit (it should be a number, instead of 0)

[elk@<my_host> bin]$ nohup ./kibana >/dev/null 2>&1 &
[4] 29926
[elk@<my_host> bin]$ grep Sig /proc/$!/status
SigQ:   1/63460
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180004202

[elk@<my_host> bin]$ nohup ./elasticsearch >/dev/null 2>&1 &
[2] 29541
[elk@<my_host> bin]$ grep Sig /proc/$!/status
SigQ:   1/63460
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000001
SigCgt: 2000000181005cce

[elk@<my_host> heartbeat-7.3.1-linux-x86_64]$ nohup ./heartbeat >/dev/null 2>&1 &
[3] 29739
[elk@<my_host> heartbeat-7.3.1-linux-x86_64]$ grep Sig /proc/$!/status
SigQ:   1/63460
SigPnd: 0000000000000000
SigBlk: fffffffe3bfa3a00
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff

[elk@itgrezlphd000 metricbeat-7.3.1-linux-x86_64]$ nohup ./metricbeat > /dev/null 2>&1 &
[1] 12077
[elk@itgrezlphd000 metricbeat-7.3.1-linux-x86_64]$ grep Sig /proc/$!/status
SigQ:   0/514512
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff


[elk@itgrezlphd000 filebeat-7.3.1-linux-x86_64]$ grep Sig /proc/$!/status
SigQ:   0/514512
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff

[elk@<my_host> apm-server-7.3.1]$ nohup ./apm-server >/dev/null 2>&1 &
[5] 30687
[elk@<my_host> apm-server-7.3.1]$ grep Sig /proc/$!/status
SigQ:   1/63460
SigPnd: 0000000000000000
SigBlk: fffffffe3bfa3a00
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff

Is my analysis right?
I have restarted metricbeats and heartbeat using disown. I will let you know if it works.
Thanks

[ea1987 (andrea)]

Hi,
disown seems to be the solution