Filebeat running as a container not displaying logs in Kibana

Elastic Stack Beats
,
1

We are running filebeat:8.1.3 as a docker container using docker-compose file in Linux RHEL machine.

We have provided the input type as filestream and provided the path for log files , but still it is not injecting any logs not displaying logs in Kibana.

Could you please help on this?

Filebeat.yml as below:

-type: filestream
id: test
enabled: true
paths:
- /var/elk/test.log

Docker-compose volume mounts:

Volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers/:/var/lib/docker/containers/:ro

2

Hi @Anagha_nambiar Please do not @ mention people directly in a thread that are not already in the thread, asking for help directly it is not community etiquette.

This is a docker mounting issue ... much like in your last question about metricbeat you will need to bind mount the host file system that you want to collect logs from.

A docker container can not see the host filesystem unless you bind mount it.

something like

    volumes:
      - type: bind
        source: $HOST/location
        target: /container/location

Perhaps you should read up on docker and storage

And Compose Syntax

3

Hi @stephenb ,
Sorry for that and thanks for letting me know. I have mentioned you because it was similar to metricbeat issue which you helped.

I have already mounted the host and container filesystem details, which is mentioned in the details I mentioned above on docker- compose file.

I am facing this issue even after that.

4

No you did not You only mounted 2 specific files not the the logs....

No...

That is not mounted or mapped to a host file location so it can not be read... Again you need to understand this from a docker perspective ...

Something like...

    volumes:
      - type: bind
        source: $HOST/location/elk/test.log
        target:  /var/elk/test.log

You need to understand docker and volume

5

Okay, thank you @stephenb

6

Hi @stephenb ,

I have created a test.yml under filebeat/modules.d folder.

Please find the test.yml below:

-type: filestream
Id: test
Enabled: true
Path: "/var/logs.log"

Then mounted this test.yml in docker-compose yml file.

But getting error like "could not create module registry for fileset"

Could you please suggest on this?

7

That is not a module... you need to put that filestream directly in the filebeat.yml

you can not add files to the modules.d directory as far as I know that is not the intention that is a predefined set of modules.

8

Okay, got you @stephenb

But we have many number of paths which needs to be monitored. It is difficult to put that directly in filebeat.yml as the file becomes huge.

Could you please suggest any workaround?

9

One container with many paths?

Or Many containers with a few paths?

to fetch all files from a predefined level of subdirectories, the following pattern can be used: /var/log/*/*.log.

You're probably going to need some form of automation/ scripting.

And apologies, I can't really offer advice on that.

10

I am actually using filebeat container to check some of the paths which is in host filesystem.

There are multiple paths under which different logs are present

11

So mount the top filesystem path / and then list the paths individually or with * just like you would have to do if filebeat was just running directly on the host...

You will still need to list the paths... no other way to do it something like.

    volumes:
      - type: bind
        source: /
        target:  /hostrootfiles/


paths: 
  - /hostrootfiles/var/*/*.log

There is no "magic" there in fact perhaps you should try running filebeat NOT in docker first get it running then convert to docker...

12

Okay, thank you for confirming.
I will try that.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.