Filebeat not reading log files from given paths

Elastic Stack Beats
1

I have created a docker image with ELK stack by using docker-compose. I have used filebeat to read log files, filebeat gives output to logstash and logstash gives outputs to elasticsearch and then finally elasticsearch gives data output to kibana dashboard. I write a yaml file called "filebeat.yml" and here is my filebeat.yml file.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /docker-containers/log/nginx/*.log
output.logstash:
  hosts: ["localhost:5044"]

When the docker image is running followings are the terminal outputs.

OTP-Filebeat     | 2019-01-09T08:15:00.099Z	INFO	instance/beat.go:400	filebeat start running.
OTP-Filebeat     | 2019-01-09T08:15:00.100Z	INFO	registrar/registrar.go:134	Loading registrar data from /usr/share/filebeat/data/registry
OTP-Filebeat     | 2019-01-09T08:15:00.100Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
OTP-Filebeat     | 2019-01-09T08:15:00.100Z	INFO	crawler/crawler.go:72	Loading Inputs: 0
OTP-Filebeat     | 2019-01-09T08:15:00.101Z	INFO	log/input.go:138	Configured paths: [/mnt/log/*.log]
OTP-Filebeat     | 2019-01-09T08:15:00.101Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
OTP-Filebeat     | 2019-01-09T08:15:00.099Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
OTP-Filebeat     | 2019-01-09T08:15:00.101Z	INFO	cfgfile/reload.go:150	Config reloader started
OTP-Filebeat     | 2019-01-09T08:15:00.101Z	INFO	cfgfile/reload.go:150	Config reloader started
OTP-Filebeat     | 2019-01-09T08:15:00.101Z	INFO	cfgfile/reload.go:205	Loading of config files completed.
OTP-Filebeat     | 2019-01-09T08:15:00.102Z	INFO	log/input.go:138	Configured paths: [/mnt/log/*.log]
OTP-Filebeat     | 2019-01-09T08:15:00.102Z	INFO	input/input.go:114	Starting input of type: log; ID: 7490297193693978066 
OTP-Filebeat     | 2019-01-09T08:15:00.102Z	INFO	cfgfile/reload.go:205	Loading of config files completed.

The Configured paths seems like different and logs are not propagate to kibana.

additional infos:

  • I have used 4 different containers to run filebeat, logstash, elasticsearch and kibana.
  • I wrote one docker-compose file to configure all docker containers.

The docker-compose.yml file as follows.

version: '2'
services:


  elasticsearch:
    container_name: OTP-Elasticsearch
    build:
      context: ./elasticsearch
      args:
        - ELK_VERSION=${ELK_VERSION}
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  filebeat:
    container_name: OTP-Filebeat
    build:
      context: ./filebeat
      args:
        - ELK_VERSION=${ELK_VERSION}
    volumes:
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/config/filebeat.yml:ro
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on: 
      - elasticsearch

  logstash:
    container_name: OTP-Logstash
    build:
      context: ./logstash
      args:
        - ELK_VERSION=${ELK_VERSION}
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5044:5044"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    links:
      - elasticsearch
    depends_on: 
      - filebeat
      - elasticsearch


  kibana:
    container_name: OTP-Kibana
    build:
      context: ./kibana
      args:
        - ELK_VERSION=${ELK_VERSION}
    volumes:
      - ./kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    networks:
      - elk
    links:
      - elasticsearch
    depends_on: 
      - elasticsearch
      - logstash

networks:
  elk:
    driver: bridge

I need to know how to configure log inputs to filebeat container correctly.

Thanks.

2

Hello @yachitha, Looking at your config and also the log output, are you sure Filebeat read the right configuration file? I would expect us to see a Configured paths containing /docker-containers

1 Like
3

@pierhugues thanks. I figure it out. The configuration file I wrote by my own is not running, instead the default filebeat.yml file running. Configured paths in default file is working that because logs are not read. Thanks again for pointing it out.

this line replaced with following.

volumes:- ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro

With this configurations I was able to fix this issue.

Important: here I tried to replace default configuration file with my own filebeat.yml file. I have faced some permission problems and after solving them.

docker-compose up --build

command build the image and now it's works fine.

If anyone faced to the same issue please follow the file structure of the elastic-filebeat docker image.
Link:
elastic/beats-docker-github

see the

beats-docker/build/filebeat/config/prospectors.d/default.yml

and

beats-docker/build/filebeat/config/filebeat.yml

you'll be able to slove the issue.

Thanks again.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.