Filebeat with Docker mounted logs handle restart duplicates

Elastic Stack Beats
1

Im currently sending logs via a mounted volume.

docker-compose.yml looks like the following

version: '3'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:6.3.0
    container_name: filebeat
    volumes:
      - '/var/log/logstash/:/mnt/log:ro'
      - './config/filebeat.yml:/usr/share/filebeat/filebeat.yml'
    restart: always
    network_mode: "host"

filebeat.yml looks like this

filebeat.prospectors:
- input_type: log
  paths:
    - /mnt/log/*.log
  json.overwrite_keys: true
  json.keys_under_root: true
  fields_under_root: true

output.logstash:
  hosts: ['localhost:5044']

Now the logs are generated by spring boot which will delete old logs after x amount of days. My concern is If I restart the machine or run a new container it will resend all the same logs.

Can I mount the filebeat "tracker file" or whatever it is called? So that it will persist reboots. Or is there another mechanism that will do the same thing?

Also did a search and this question has been asked before.. However couldnt really work out a solution based on the answers
https://discuss.elastic.co/t/filebeat-persist-processed-logs-if-docker-container-crashes/60089

2

I would recommend you to create a volume for the registry file which is inside the Filebeat data directory as you suggested. Point the data directory to this volume and mount it into your Filebeat container. So if the Filebeat container stops you can just start one up again and continue reading.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.