# FileBeat not harvesting new logs in the provided path

**URL:** <https://discuss.elastic.co/t/filebeat-not-harvesting-new-logs-in-the-provided-path/250485>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 30, 2020, 10:49am UTC](https://discuss.elastic.co/t/filebeat-not-harvesting-new-logs-in-the-provided-path/250485 "2020-09-30T10:49:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![BTH.S3](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@BTH.S3](https://discuss.elastic.co/u/BTH.S3)\
**Post date:** [September 30, 2020, 10:49am UTC](https://discuss.elastic.co/t/filebeat-not-harvesting-new-logs-in-the-provided-path/250485/1 "2020-09-30T10:49:50Z")

</div>

FileBeat has been running as a docker container started by docker-compose file and already mapped the volume for filebeat.yml.

I've updated the configuration in filebeat.yml by setting input and output as Elasticsearch.

```auto
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
                # - "/usr/share/filebeat/logs/*.log"
                #- "/deploy/tomcat/logs/*.access*.log"
  
    output.elasticsearch:
      hosts: ["deploy-01.com:9200"]
      username: "elastic"
      password: "changeme"
      indices:
        - index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
        - index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"
          when.contains:
            message: "WARN"
        - index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
          when.contains:
            message: "ERR"

    setup.kibana:
      host: "deploy-01.com:5601"
      username: "elastic"
      password: "changeme"

```

It turns out that the FileBeat only harvest once for the first time after the fresh installation. Although the logs in the provided directory always create every day.

Checked for the permission of the directory and files  
docker logs \<filebeat\_container\> only gives these

`2020-09-30T10:29:24.087Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":360},"total":{"ticks":1560,"time":{"ms":7},"value":1560},"user":{"ticks":1200,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"c038389c-cdac-4f29-bff1-69e94bda3d36","uptime":{"ms":3810205}},"memstats":{"gc_next":8119184,"memory_alloc":4486864,"memory_total":50182328},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":7.27,"15":7.6,"5":7.56,"norm":{"1":0.2272,"15":0.2375,"5":0.2363}}}}}}`

Restart the container, re-create index pattern doesn't help.

Here are the logs in the directory, created every day

```auto
    -rw-rw-r-- 1 deploy deploy 271 Sep 27 15:30 deploy.com.access.2020-09-27.log
    -rw-rw-r-- 1 deploy deploy 70876 Sep 28 18:28 deploy.com.access.2020-09-28.log
    -rw-rw-r-- 1 deploy deploy 295 Sep 29 04:20 deploy.com.access.2020-09-29.log
    -rw-rw-r-- 1 deploy deploy 3268 Sep 30 11:05 deploy.com.access.2020-09-30.log

```

Kibana dashboard doesn't show any new logs

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a7f2c8126c1860b3f7298e43133ecbf351ba11a.png)

Please help,  
Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 28, 2020, 12:49pm UTC](https://discuss.elastic.co/t/filebeat-not-harvesting-new-logs-in-the-provided-path/250485/2 "2020-10-28T12:49:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.