FileBeat not harvesting new logs in the provided path

FileBeat has been running as a docker container started by docker-compose file and already mapped the volume for filebeat.yml.

I've updated the configuration in filebeat.yml by setting input and output as Elasticsearch.

    filebeat.inputs:
      - type: log
        enabled: true
        paths:
                # - "/usr/share/filebeat/logs/*.log"
                #- "/deploy/tomcat/logs/*.access*.log"
  
    output.elasticsearch:
      hosts: ["deploy-01.com:9200"]
      username: "elastic"
      password: "changeme"
      indices:
        - index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
        - index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"
          when.contains:
            message: "WARN"
        - index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
          when.contains:
            message: "ERR"

    setup.kibana:
      host: "deploy-01.com:5601"
      username: "elastic"
      password: "changeme"

It turns out that the FileBeat only harvest once for the first time after the fresh installation. Although the logs in the provided directory always create every day.

Checked for the permission of the directory and files
docker logs <filebeat_container> only gives these

2020-09-30T10:29:24.087Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":360},"total":{"ticks":1560,"time":{"ms":7},"value":1560},"user":{"ticks":1200,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"c038389c-cdac-4f29-bff1-69e94bda3d36","uptime":{"ms":3810205}},"memstats":{"gc_next":8119184,"memory_alloc":4486864,"memory_total":50182328},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":7.27,"15":7.6,"5":7.56,"norm":{"1":0.2272,"15":0.2375,"5":0.2363}}}}}}

Restart the container, re-create index pattern doesn't help.

Here are the logs in the directory, created every day

    -rw-rw-r-- 1 deploy deploy    271 Sep 27 15:30 deploy.com.access.2020-09-27.log
    -rw-rw-r-- 1 deploy deploy  70876 Sep 28 18:28 deploy.com.access.2020-09-28.log
    -rw-rw-r-- 1 deploy deploy    295 Sep 29 04:20 deploy.com.access.2020-09-29.log
    -rw-rw-r-- 1 deploy deploy    3268 Sep 30 11:05 deploy.com.access.2020-09-30.log

Kibana dashboard doesn't show any new logs

Please help,
Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.