Filebeat folder structured not correct for version 7.10

Elastic Stack Beats
#1

filebeat bin file location in the service file declared might be in the wrong location for debian platform. after moving the filebeat bin file (located in /usr/ share/filebeat/) to /usr/share/filebeat/bin/ folder everything works.

the service file states
ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS

which expects it to be in the bin folder but the default installation comes one directory up from the bin folder.

#2

Hey @kbirhan welcome to discuss :slight_smile:

The official packages for debian, downloaded from https://www.elastic.co/downloads/beats/filebeat, have the filebeat binary in /usr/share/filebeat/bin/filebeat:

$ dpkg -c filebeat-7.10.0-amd64.deb | egrep filebeat$
-rwxr-xr-x 0/0            5080 2020-11-09 21:01 ./etc/init.d/filebeat
-rwxr-xr-x 0/0             325 2020-11-09 21:01 ./usr/bin/filebeat
-rwxr-xr-x 0/0        99345344 2020-11-09 20:59 ./usr/share/filebeat/bin/filebeat

They also install a wrapper in /usr/bin/filebeat.

What packages are you using?

#3

i am using 7.10.0. following the instruction on kibana dashboard to add s3 based logs (http://localhost:5601/app/home#/tutorial/awsLogs).
this is is the command given on the instruction to install filbeat on Debian.
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-amd64.deb.

when i look into the log files the only Error log output is the below line
ERROR fileset/modules.go:127 Not loading modules. Module directory not found: /usr/share/filebeat/module.
no idea why it is not seeing the module folder even though it's there. the behaviour is the same when installing from the link you provided.

#4

These instructions work for me.

But you could be affected by this bug, specific to AWS module: https://github.com/elastic/beats/issues/22608
It doesn't have anything to do with the folder structure in any case.

Yes, this package has the binary in /usr/share/filebeat/bin/filebeat, as expected by the systemd unit file.

Ok, this path is not related to the binary, but still, this path is provided by the filebeat package:

$ dpkg -S /usr/share/filebeat/module/
filebeat: /usr/share/filebeat/module

$ dpkg -c /tmp/filebeat-7.10.0-amd64.deb | grep "/usr/share/filebeat/module" | head
drwxr-xr-x 0/0               0 2020-11-09 21:01 ./usr/share/filebeat/module/
drwxr-xr-x 0/0               0 2020-11-09 21:01 ./usr/share/filebeat/module/squid/
-rw-r--r-- 0/0             163 2020-11-09 20:56 ./usr/share/filebeat/module/squid/README.md
drwxr-xr-x 0/0               0 2020-11-09 21:01 ./usr/share/filebeat/module/squid/log/
....
....

Can you try to reinstall the package? Are you installing it with dpkg as described in the instructions you mention?

What linux distribution and on what version are you using?

#5

Hi Jamie I think i might have been affected by the bug you mentioned, after applying the fix its up and running now. my system is Ubuntu 20.04. but still not able to pull data from s3 bucket. followed this instruction([https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack]) to set up s3 bucket for cloud trail logs and sqs queue. data gets pulled on my mac though.
ERROR [input.s3] s3/collector.go:107 SQS ReceiveMessageRequest failed: EC2RoleRequestError: no EC2 instance role found

#7

Hi Jamie i wanted to let you know that i was able to resolve the problem. For some reason, it was not able to pick the aws credentials from the .aws directory. I had to put the credentials in the aws.yml file directly to resolve the problems. Also, another way that worked was specifying
var.shared_credential_file to the /home/{your-user-name}/.aws/credentials. it should have looked in that directory if not specified but for some reason, it will not work unless explicitly specified.

#8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.