Filebeat service hangs on first start amazon ec2

(Brice Miramont) #1


I have a java app on elastic beanstalk.
I have a ./ebextensions/filebeat.config deployed to my ec2 instances upon EB deployement.

Problem is : the first time the filebeat process is started, it hangs. I have to execute sudo /etc/init.d/filebeat restart to have the deployement resumes.
the command I use for starting the process is : /etc/init.d/filebeat start

This happens when the command is in the command block, or in a delayed job in the .config file.
here is the point where it hangs :
[2016-04-05T17:28:06.251Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/] : Starting activity...
[2016-04-05T17:28:06.755Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/] : Completed activity.
[2016-04-05T17:28:06.755Z] INFO [16221] - [Application update/AppDeployStage1/AppDeployPostHook/] : Starting activity...

my script:
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
/etc/init.d/filebeat start

After that, all the following deployements works normally.

thank you.

(Brice Miramont) #2

same behavior with topbeat...

(Tudor Golubenco) #3

What operating system are you using? Does it have systemd?

(Brice Miramont) #4

it's an Amazon AMI :
64bit Amazon Linux 2015.09 v2.0.7 running Tomcat 8 Java 8

I don't think there is systemd

(Tudor Golubenco) #5

I see, we did get some other reports regarding this. Most likely the problem is caused by go-daemon, the supervisor that we use. We only use it to put the process in background, so you could replace it fairly easily with the daemon tool by editing the init script.

I think I have a grasp now on why go-daemon doesn't do daemonization correctly. If I send you custom filebeat-god binary, would you be willing to give it a try to see if it fixes the issue?

(Brice Miramont) #6

sure, I'll give it a try

(Tudor Golubenco) #7

Thanks, are you on 32 bits or 64 bits? Or do you have gcc installed on a machine? It might be easier if you compile it yourself, I'll just give you the right commands.

(Brice Miramont) #8

it's 64 bits but I doubt there is gcc on it : 64bit Amazon Linux 2015.09

no gcc but that can be installed via yum

(Tudor Golubenco) #9

No need then, I'll give you a binary. Let me just find a place to upload it.

(Brice Miramont) #10

ok great, and please tell me where to put the file then

(Tudor Golubenco) #11

Ok, here is the file:

It's a binary build for Linux 64 bit, so you should be able to execute it with -h just to make sure that the download worked fine and the file is executable.

If you test with Filebeat you need to copy it over /usr/bin/filebeat-god (just replace that file, make a backup if you want). If you test with Topbeat, you need to replace /usr/bin/topbeat-god.

Let me know if this seems to solve the issue.

(Brice Miramont) #12

ok, i'll give it a try

(Brice Miramont) #13

hi, I'm trying to replace the file but it is a bit tricky with AWS elastic beanstalk deployement options.

I'll tell when you when it's ok but can you provide me with an rpm with the new god file ? that way I can deploy the whole package on a new platform and thus I'll be able to validate that it's working.

thank you

(Tudor Golubenco) #14

Let me try to produce one.

(Brice Miramont) #15

great !!

(Brice Miramont) #16

service is running on one instance but not on another one. even when I start manually.
I'll put the log and post them here

(Brice Miramont) #17

ok, strange thing.
here is the output of my command:

[ec2-user@ip-172-32-10-167 ~]$ sudo service topbeat status
topbeat-god dead but pid file exists

but he service is working: 
2016-04-14T09:24:10Z DBG  output worker: publish 107 events
2016-04-14T09:24:10Z DBG  Try to publish 107 events to logstash with window size 106
2016-04-14T09:24:10Z DBG  106 events out of 107 events sent to logstash. Continue sending ...
2016-04-14T09:24:10Z DBG  Try to publish 1 events to logstash with window size 106
2016-04-14T09:24:10Z DBG  1 events out of 1 events sent to logstash. Continue sending ...
2016-04-14T09:24:10Z DBG  send completed

(Brice Miramont) #18

I'll deploy to a fresh environment when I'll have the rpm and we'll have a confirmation that it works

(Tudor Golubenco) #19

Yeah, it's best to wait for the RPM and try on a fresh system I think.

(Brice Miramont) #20

there is several processes running so no wonder the beahvior is strange :slight_smile:
[ec2-user@ip-172-32-10-167 ~]$ ps -edf | grep beat
root 2693 1 0 09:03 ? 00:00:00 filebeat-god -r / -n -p /var/run/ -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 2694 2693 0 09:03 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 2736 1 0 09:03 ? 00:00:00 topbeat-god -r / -n -p /var/run/ -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 2737 2736 0 09:03 ? 00:00:04 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3100 1 0 09:04 ? 00:00:00 topbeat-god -r / -n -p /var/run/ -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3103 3100 0 09:04 ? 00:00:04 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3650 1 0 09:18 ? 00:00:00 filebeat-god -r / -n -p /var/run/ -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 3651 3650 0 09:18 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 3696 1 0 09:18 ? 00:00:00 topbeat-god -r / -n -p /var/run/ -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 3697 3696 0 09:18 ? 00:00:01 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 4181 1 0 09:20 ? 00:00:00 filebeat-god -r / -n -p /var/run/ -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4182 4181 0 09:20 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4259 1 0 09:21 ? 00:00:00 filebeat-god -r / -n -p /var/run/ -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4260 4259 0 09:21 ? 00:00:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root 4391 1 0 09:23 ? 00:00:00 topbeat-god -r / -n -p /var/run/ -- /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
root 4392 4391 0 09:23 ? 00:00:00 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml
ec2-user 4531 2507 0 09:26 pts/0 00:00:00 grep --color=auto beat