Need to gather the host logs , filebeat running as a container

Elastic Stack Beats
,
1

Hello Team,

I am new to ELK and Filebeat.
I have ELK stack running on one server using docker-compose.
I have Filebeat running on another server as docker container 7.14.1.

I have configured filebeat-docker.yml and I can see some logs at the Kibana dashboard which are coming within the containers.

My question here is as I am running filebeat as a container, however I wanted to collect the application and system logs which are running on host not on the container. As the application running on host on which filebeat running as a docker container

Is it possible? if yes, can you please guide/suggest me how to achieve it?

Thank you in advance.
Sabil.

2

Welcome to our community! :smiley:

Generally you would install Filebeat directly on the host, that way you can monitor the host and any other containers it runs. Running Filebeat in a container and monitoring the underlying host is possible, you just need to make sure it has the relevant local filesystem points mounted inside the container.

3

@warkolm Thank you so much! :slight_smile:

Let me try mounting the log path as volume in the filebeat container. Just to cross check, I will be executing the following command to mount the volumes.

Hopefully is the correct way to do it. Please correct me if I am wrong.

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="/var/log/apache2/error.log:/var/log/apache2/error.log:ro"
  --volume="/var/log/apache2/access.log:/var/log/apache2/xyz.log:ro"
  --volume="/var/log/apache2/dispatcher.log:/var/log/apache2/abc.log:ro"

Thank you.
Sabil.

1 Like
4

@warkolm I can see the volumes have been mounted by executing the above command. However, I am not getting that data in the Kibana dashboard.

I am referring log.file.path in kibana dashboard. I can see only /var/lib/containers.

Following info I am getting in filebeat logs.
INFO [input.harvester] log/harvester.go:340 File is inactive. Closing because close_inactive of 5m0s reached.

Any suggestion and help would be appreciated.

Thank you.
Sabil.

5

What does your Filebeat config look like?

6

Here is a filebeat-docker.yml file.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/error.log
    - /var/log/apache2/access.log
    - /var/log/apache2/dispatcher.log
    - /var/log/aem/error.log
  include_lines: ['^ERR', '^WARN']

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
7

You're probably better off using the Apache module if you can - Apache module | Filebeat Reference [7.14] | Elastic

8

Sure, let me try adding apache module to config.

Thank you!

9

@warkolm Sorry, but still getting the same.

here is config file.

filebeat.config:
- modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

- module: apache
  access:
    enabled: true
    var.paths: ["/path/to/log/apache/access.log*"]
  error:
    enabled: true
    var.paths: ["/path/to/log/apache/error.log*"]

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/error.log
    - /var/log/apache2/access.log
    - /var/log/apache2/dispatcher.log
    - /var/log/aem/error.log
  include_lines: ['^ERR', '^WARN']

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'

thank you.

10

@warkolm Any suggestion would be highly appreciated. Thank you.

11

Can you try running with debug and seeing if that says anything?

12

@warkolm I am running the following command and not getting debug option while executing the following command.

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker-public.docker.devstack.vwgroup.com/elastic/filebeat:7.14.1 -e -strict.perms=false
13

@warkolm I am able to get the logs on kinbana dashboard.

Can you please suggest on how to filter the log data ?

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.