How can I send custom logs in a specific location to filebeat running inside docker

Elastic Stack Beats
,
1

I am new to filebeat and elk. I am trying to send custom logs using filebeat to Elasticsearch directly.Both the elk stack and filebeat are running inside docker containers.. The custom logs are in the folder home/username/docker/hello.log. Here is my filebeat.yml file:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/raju/elk/docker/*.log
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: ["my_ip:9200"]

And here is my custom log file:

This is a custom log file 
Sending logs to elastic search

And these are the commands using which I am using to run filebeat:

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:8.5.3 filebeat -e --strict.perms=false

When i use the above commands to run filebeat I can see the logs of the docker containers on my kibana dashboard. But I am struggling on how to make filebeat to read my custom logs from the specified location above and show me the lines inside the log file.

Anyhelp would be appreciated.

2

If those files are on the host system you will need to mount those to the filebeat container so the filebeat container can access them

By default a docker container is isolated from the host filesystem.

Perhaps you should read about docker accessing host filesystems.here

This mounted the filebeat.yml

--volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro

3

@stephenb Thank you for valuable reply. I did mount the current directory as a volume to docker container running filebeat but still I can't access the log file and kibana shows no index pattern. Maybe I am not sure if log file needs a certain format or not.
--volume="$(pwd)/hello.log:/usr/share/filebeat/hello.log:ro

I guess you mean like this. Path inside docker can be changed I am sure but it has no effect as it is not reading the lines inside my log file

4

This is more of a docker issue than a filebest issue.

I would mount the file system / directory and then exec into the filebeat docker container and see if you can see those files.... Once you can see those files ... set up filebeat correctly.

Until file beat can access those files, it clearly won't be able to ship them to Kibana.

Perhaps you should try filebeat outside of docker first. Get that working correct.... And then try it with docker.

5 
2022-12-29T15:37:35.532Z	INFO	instance/beat.go:645	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2022-12-29T15:37:35.672Z	INFO	instance/beat.go:653	Beat ID: cf45bdc7-e611-4f0d-a384-89d6853908ab
2022-12-29T15:37:35.673Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2022-12-29T15:37:35.673Z	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "cf45bdc7-e611-4f0d-a384-89d6853908ab"}}}
2022-12-29T15:37:35.674Z	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "1da173a9e716715a7a54bb3ff4db05b5c24fc8ce", "libbeat": "7.10.1", "time": "2020-12-04T23:27:17.000Z", "version": "7.10.1"}}}
2022-12-29T15:37:35.674Z	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.12"}}}
2022-12-29T15:37:35.675Z	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-12-29T08:33:31Z","containerized":false,"name":"0c5e284a3f33","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"5.15.0-56-generic","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"d624e9f79ced4be589fadca86f6cdcf0"}}}
2022-12-29T15:37:35.708Z	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-12-29T15:37:34.190Z"}}}
2022-12-29T15:37:35.708Z	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.1
2022-12-29T15:37:35.708Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.10.1' as ILM is enabled.
2022-12-29T15:37:35.709Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://192.168.1.38:9200
2022-12-29T15:37:35.709Z	INFO	[publisher]	pipeline/module.go:113	Beat name: 0c5e284a3f33
2022-12-29T15:37:35.710Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2022-12-29T15:37:35.710Z	INFO	instance/beat.go:455	filebeat start running.
2022-12-29T15:37:35.842Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2022-12-29T15:37:35.842Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2022-12-29T15:37:35.843Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2022-12-29T15:37:35.843Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 1
2022-12-29T15:37:35.843Z	INFO	log/input.go:157	Configured paths: [/home/raju/elk/docker/hello.log]
2022-12-29T15:37:35.843Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 273078644940147100)
2022-12-29T15:37:35.844Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 1
2022-12-29T15:37:35.844Z	INFO	cfgfile/reload.go:164	Config reloader started
2022-12-29T15:37:35.844Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2022-12-29T15:37:38.708Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2022-12-29T15:38:05.714Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":32}},"total":{"ticks":180,"time":{"ms":187},"value":180},"user":{"ticks":150,"time":{"ms":155}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":30234}},"memstats":{"gc_next":17209040,"memory_alloc":9424872,"memory_total":44276904,"rss":69820416},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":0.59,"15":0.81,"5":0.82,"norm":{"1":0.1475,"15":0.2025,"5":0.205}}}}}}
2022-12-29T15:38:35.729Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":1}},"total":{"ticks":190,"time":{"ms":6},"value":190},"user":{"ticks":160,"time":{"ms":5}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":60249}},"memstats":{"gc_next":17209040,"memory_alloc":9662192,"memory_total":44514224,"rss":-1777664},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.68,"15":0.82,"5":0.82,"norm":{"1":0.17,"15":0.205,"5":0.205}}}}}}
2022-12-29T15:39:05.716Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":9}},"total":{"ticks":200,"time":{"ms":11},"value":200},"user":{"ticks":160,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":90235}},"memstats":{"gc_next":17209040,"memory_alloc":10084008,"memory_total":44936040,"rss":-950272},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.58,"15":0.81,"5":0.79,"norm":{"1":0.145,"15":0.2025,"5":0.1975}}}}}}
2022-12-29T15:39:35.781Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":5}},"total":{"ticks":220,"time":{"ms":29},"value":220},"user":{"ticks":180,"time":{"ms":24}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":120233}},"memstats":{"gc_next":17836992,"memory_alloc":10144576,"memory_total":45168992},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.35,"15":0.78,"5":0.71,"norm":{"1":0.0875,"15":0.195,"5":0.1775}}}}}}
2022-12-29T15:40:05.715Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":37}},"total":{"ticks":300,"time":{"ms":80},"value":300},"user":{"ticks":220,"time":{"ms":43}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":150237}},"memstats":{"gc_next":17836992,"memory_alloc":9252552,"memory_total":45503528},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.28,"15":0.76,"5":0.66,"norm":{"1":0.07,"15":0.19,"5":0.165}}}}}}
2022-12-29T15:40:35.715Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":3}},"total":{"ticks":310,"time":{"ms":9},"value":310},"user":{"ticks":230,"time":{"ms":6}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":180233}},"memstats":{"gc_next":17836992,"memory_alloc":9590776,"memory_total":45841752},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.71,"15":0.77,"5":0.71,"norm":{"1":0.1775,"15":0.1925,"5":0.1775}}}}}}
2022-12-29T15:41:05.715Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":5}},"total":{"ticks":330,"time":{"ms":11},"value":330},"user":{"ticks":240,"time":{"ms":6}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":210233}},"memstats":{"gc_next":17836992,"memory_alloc":9917784,"memory_total":46168760},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.43,"15":0.75,"5":0.64,"norm":{"1":0.1075,"15":0.1875,"5":0.16}}}}}}
2022-12-29T15:41:35.712Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":1}},"total":{"ticks":330,"time":{"ms":8},"value":330},"user":{"ticks":240,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"de713df3-878c-4a42-9d81-5ae685da0552","uptime":{"ms":240234}},"memstats":{"gc_next":17836992,"memory_alloc":10332720,"memory_total":46583696},"runtime":{"goroutines":22}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.46,"15":0.74,"5":0.63,"norm":{"1":0.115,"15":0.185,"5":0.1575}}}}}}

These are the logs from filebeat I guess it is reading the file.

6

There should be some more logs about how many it published, shipped and acked....

Also, it'll only read the file once so it already read it once It's not going to read it again unless you destroy the container and recreate it because it keeps track of what it's read.

See those long lines about the registry that's keeping track of what it already read. So running filebeat over and over on the same log file does not work. That's a fundamental concept of filebeat unless you clean out the registry or more log lines or added to your file.

You could try a appending some more lines to your log file and see what happens.

Again, if you're just beginning I would probably run file beat first, not in docker. You're actually trying to make it do something that's a little more advanced.

Also, why such an old version? 7.10

You should be running 7.17 at least

7

I have added the full output of filebeat by editing my previous reply. I can see the mount is successful inside docker.

sh-4.2# ls
LICENSE.txt  NOTICE.txt  README.md  data  fields.yml  filebeat	filebeat.reference.yml	filebeat.yml  kibana  logs  logstash.log  logstash-json.log~  module  modules.d
sh-4.2#

Here logstash.log is the log file I added. It was working with this command:

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:8.5.3 filebeat -e --strict.perms=false

As it is looking for running containers on my system and is logging everything to my kibana dashboard. But I am not sure why its not working with a custom log file. Maybe you are right I need to test it by installing filebeat on my local system.

8

I appreciate your patience. But I am not sure maybe I need to give the path of log file inside docker after mounting in filebeat.yml instead of giving localhost path like home/raju/elk/docker/.log to path inside docker /usr/share/filbeat/.log. I am not sure maybe that it would pick up.

9

Yes, it has to be path in the docker container

I'm trying to give you suggestions. I suggested that you exec into the docker container once it is running and look for the path for your log file. That is the path that has to be in the yml file that filebeat uses

10

That should be the path in the docker container

Again, this is an understanding about how docker works Perhaps spend a little more time understanding the interaction between docker and the host

11

I did show the output of docker container above by exec into the container. I see both the filebeat.yml and log file I added from localhost are there and have some content that I specified from my localhost.

12

Here is the output of the filebeat.yml inside docker container.

sh-4.2# ls
LICENSE.txt  NOTICE.txt  README.md  data  fields.yml  filebeat	filebeat.reference.yml	filebeat.yml  kibana  logs  logstash-json.log  module  modules.d
sh-4.2# cat filebeat.yml
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.inputs:
- type: log
  paths:
    - /home/raju/elk/docker/hello.log
#filebeat.autodiscover:
 # providers:
  #  - type: docker
   #   hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:192.168.1.38:9200}'
# username: '${ELASTICSEARCH_USERNAME:}'
#  password: '${ELASTICSEARCH_PASSWORD:}'
sh-4.2#
13

Shouldn't that be the path inside the docker container ....not the host path

What does your mount look for that Please show

14

Here is how I am mounting the file. Please do not mind the log file names as I was experimenting with different files.

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="$(pwd)/hello.log:/usr/share/filebeat/hello.log:ro" \
  docker.elastic.co/beats/filebeat:7.10.1 filebeat -e --strict.perms=false  \
  -E output.elasticsearch.hosts=["192.168.1.38:9200"]
15

You are absolutely right. I needed to change the path inside filebeat.yml from this :

    - /home/raju/elk/docker/hello.log

To this after mounting volume.

./hello.log

Thank you so much. You are the best. You saved me a headache of two days and saved me a lot of time.

1 Like
16

One last request before you go. Is there a way I can set a filter so I can only see warnings and error messages from my log files instead of having logged everthing.

17

No Problem... perhaps slow down a bit and think about the container's perspective...

Many posts ago I said...

Ok now... keep going!

1 Like
18

Thank you so much @stephenb. I cannot thank enough for your valueable time.

19

That question is not clear...

The App Logs... filter with processors, ingest pipelines

The Filebeat Logs .... filter with logging levels

Which logs do you want to filter?

And the answer is yes... but perhaps you should take a look at the filebeat documentation and see if you can figure it out :slight_smile:

1 Like
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.