Filebeat service hangs on first start amazon ec2

Elastic Stack Beats
1

Hi,

I have a java app on elastic beanstalk.
I have a ./ebextensions/filebeat.config deployed to my ec2 instances upon EB deployement.

Problem is : the first time the filebeat process is started, it hangs. I have to execute sudo /etc/init.d/filebeat restart to have the deployement resumes.
the command I use for starting the process is : /etc/init.d/filebeat start

This happens when the command is in the command block, or in a delayed job in the .config file.
here is the point where it hangs :
[2016-04-05T17:28:06.251Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/03monitor_pids.sh] : Starting activity...
[2016-04-05T17:28:06.755Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/03monitor_pids.sh] : Completed activity.
[2016-04-05T17:28:06.755Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/99_restart_delayed_job.sh] : Starting activity...

my script:
file:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_restart_delayed_job.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
/etc/init.d/filebeat start

After that, all the following deployements works normally.

thank you.

2

same behavior with topbeat...

3

What operating system are you using? Does it have systemd?

4

it's an Amazon AMI :
64bit Amazon Linux 2015.09 v2.0.7 running Tomcat 8 Java 8

I don't think there is systemd

5

I see, we did get some other reports regarding this. Most likely the problem is caused by go-daemon, the supervisor that we use. We only use it to put the process in background, so you could replace it fairly easily with the daemon tool by editing the init script.

I think I have a grasp now on why go-daemon doesn't do daemonization correctly. If I send you custom filebeat-god binary, would you be willing to give it a try to see if it fixes the issue?

6

sure, I'll give it a try

7

Thanks, are you on 32 bits or 64 bits? Or do you have gcc installed on a machine? It might be easier if you compile it yourself, I'll just give you the right commands.

8

it's 64 bits but I doubt there is gcc on it : 64bit Amazon Linux 2015.09

no gcc but that can be installed via yum

9

No need then, I'll give you a binary. Let me just find a place to upload it.

10

ok great, and please tell me where to put the file then

11

Ok, here is the file: https://beats-nightlies.s3.amazonaws.com/god-linux-64

It's a binary build for Linux 64 bit, so you should be able to execute it with -h just to make sure that the download worked fine and the file is executable.

If you test with Filebeat you need to copy it over /usr/bin/filebeat-god (just replace that file, make a backup if you want). If you test with Topbeat, you need to replace /usr/bin/topbeat-god.

Let me know if this seems to solve the issue.

12

ok, i'll give it a try

13

hi, I'm trying to replace the file but it is a bit tricky with AWS elastic beanstalk deployement options.

I'll tell when you when it's ok but can you provide me with an rpm with the new god file ? that way I can deploy the whole package on a new platform and thus I'll be able to validate that it's working.

thank you

14

Let me try to produce one.

15

great !!

16

service is running on one instance but not on another one. even when I start manually.
I'll put the log and post them here

17

ok, strange thing.
here is the output of my command:

[ec2-user@ip-172-32-10-167 ~]$ sudo service topbeat status
topbeat-god dead but pid file exists

but he service is working: 
2016-04-14T09:24:10Z DBG  output worker: publish 107 events
2016-04-14T09:24:10Z DBG  Try to publish 107 events to logstash with window size 106
2016-04-14T09:24:10Z DBG  106 events out of 107 events sent to logstash. Continue sending ...
2016-04-14T09:24:10Z DBG  Try to publish 1 events to logstash with window size 106
2016-04-14T09:24:10Z DBG  1 events out of 1 events sent to logstash. Continue sending ...
2016-04-14T09:24:10Z DBG  send completed
18

I'll deploy to a fresh environment when I'll have the rpm and we'll have a confirmation that it works

19

Yeah, it's best to wait for the RPM and try on a fresh system I think.

20

yep.
there is several processes running so no wonder the beahvior is strange :slight_smile:
[ec2-user@ip-172-32-10-167 ~]$ ps -edf | grep beat
root 2693 1 0 09:03 ? 00:00:00 filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 2694 2693 0 09:03 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 2736 1 0 09:03 ? 00:00:00 topbeat-god -r / -n -p /var/run/topbeat.pid -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 2737 2736 0 09:03 ? 00:00:04 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3100 1 0 09:04 ? 00:00:00 topbeat-god -r / -n -p /var/run/topbeat.pid -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3103 3100 0 09:04 ? 00:00:04 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3650 1 0 09:18 ? 00:00:00 filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 3651 3650 0 09:18 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 3696 1 0 09:18 ? 00:00:00 topbeat-god -r / -n -p /var/run/topbeat.pid -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3697 3696 0 09:18 ? 00:00:01 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 4181 1 0 09:20 ? 00:00:00 filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4182 4181 0 09:20 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4259 1 0 09:21 ? 00:00:00 filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4260 4259 0 09:21 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4391 1 0 09:23 ? 00:00:00 topbeat-god -r / -n -p /var/run/topbeat.pid -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 4392 4391 0 09:23 ? 00:00:00 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
ec2-user 4531 2507 0 09:26 pts/0 00:00:00 grep --color=auto beat