Help me - How can I configure firebeat to read log files from log folder

Elastic Stack Beats
1

I have an app that write log to .log file to a log folder that mounted from container to host.

I'm very new to ELK overall so I want filebeat to read .log files generated in that log folder. Here is my filebeat config file

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0600
  
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
- type: log
  paths:
    - /media/HDD10TB/log/*.log

processors:
# The following example enriches each event with the machine's local time zone
# offset from UTC.
#      
- add_locale:
      format: abbreviation
#
# The following example enriches each event with host metadata.
#      
- add_host_metadata: ~    

output.logstash:
   hosts: ["logstash-01:5400"]

when I go to index management on Kibana I see 0 Docs count. What is the problem here? How can I solve it.?
Thanks

image
image1366×372 29.5 KB

Here is my docker-compose.yml

version: '3.8'
services:
    es-node-01:
      # network_mode: elastic
        container_name: es-node-01
        ports:
            - '9200:9200'
            - '9300:9300'
        environment:
            - discovery.type=single-node
        image: 'docker.elastic.co/elasticsearch/elasticsearch:7.17.0'
        networks:
            - elastic
    kibana-01:
      # network_mode: elastic
        container_name: kibana-01
        ports:
            - '5601:5601'
        environment:
            - 'ELASTICSEARCH_HOSTS=http://es-node-01:9200'
        image: 'docker.elastic.co/kibana/kibana:7.17.0' 
        healthcheck:
            test: ["CMD", "curl", "-f", "kibana-01:5601"]
            interval: 50s
            timeout: 50s
            retries: 5
        depends_on:
            - es-node-01
        networks:
            - elastic 

    logstash:
        container_name: logstash-01
        volumes:
            - logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro 
            - logstash.yaml:/usr/share/logstash/config/logstash.yml:ro 
        image: 'docker.elastic.co/logstash/logstash:7.17.0'
        depends_on:   
             kibana-01:
               condition: service_healthy  
        networks:
            - elastic 


    filebeat:
        user: root
        container_name: filebeat-01
        command: --strict.perms=false
        volumes:
            - filebeat.docker.yaml:/usr/share/filebeat/filebeat.yml:ro
            - /var/lib/docker/containers:/var/lib/docker/containers:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
        image: 'docker.elastic.co/beats/filebeat:7.17.0' 
        depends_on:   
             kibana-01:
               condition: service_healthy
        networks:
            - elastic              
networks:
  elastic:
    name: elastic
2

Hello and welcome,

If you are running filebeat in a docker container, the path with your logs needs to be accessible to your container.

In your filebeat you have this configuration:

filebeat.inputs:
- type: log
  paths:
    - /media/HDD10TB/log/*.log

But in your docker compose you are not bind mounting the /media path to your filebeat container.

    filebeat:
        user: root
        container_name: filebeat-01
        command: --strict.perms=false
        volumes:
            - filebeat.docker.yaml:/usr/share/filebeat/filebeat.yml:ro
            - /var/lib/docker/containers:/var/lib/docker/containers:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro

You need to make the log path accessible to your container, this is related to docker, not filebeat itself.

3

Thank you very much for your response. I will try to mount the log file to docker and will let you know if it works

5

Hi, there is logging.level line in filebeat.yml so is there a config to track all log level including Info, Error, Debug and Critical ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.