# Filebeat6.2.1 logs are not collected intermittently

**URL:** <https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 29, 2019, 7:27am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607 "2019-10-29T07:27:27Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tom\_Chen1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_chen1/32/56752_2.png) [@Tom\_Chen1](https://discuss.elastic.co/u/Tom_Chen1)\
**Post date:** [October 29, 2019, 7:27am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/1 "2019-10-29T07:27:28Z")

</div>

### I don't see any changes in the log for a long time

```auto
2019-10-29T07:25:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2500,"time":2503},"total":{"ticks":8140,"time":8145,"value":8140},"user":{"ticks":5640,"time":5642}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":24780010}},"memstats":{"gc_next":4881376,"memory_alloc":4255320,"memory_total":1641005672}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.3,"15":0.03,"5":0.13,"norm":{"1":0.15,"15":0.015,"5":0.065}}}}}}
2019-10-29T07:26:08.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2510,"time":2510},"total":{"ticks":8150,"time":8157,"value":8150},"user":{"ticks":5640,"time":5647}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":24810010}},"memstats":{"gc_next":5866896,"memory_alloc":4138808,"memory_total":1643016232}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.26,"15":0.04,"5":0.14,"norm":{"1":0.13,"15":0.02,"5":0.07}}}}}}
-----
2019-10-29T07:35:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2560,"time":2565},"total":{"ticks":8340,"time":8346,"value":8340},"user":{"ticks":5780,"time":5781}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25380010}},"memstats":{"gc_next":5852736,"memory_alloc":3530312,"memory_total":1681225912}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.01,"15":0.01,"5":0.04,"norm":{"1":0.005,"15":0.005,"5":0.02}}}}}}

```

### But I have the log files I need to collect

```auto
/var/www/webapp/logs/A/B-FILES:
total 60K
-rw-r--r-- 1 root root 50K Oct 29 00:34 B-FILES.log

```

### My configuration file is

```auto
sh-4.2$ sudo cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/applog/**/*.log
    - /var/www/webapp/logs/**/*.log
  multiline:
    pattern: '^\d+-\d+-\d+ \d+:\d+:\d+\.\d+|^\d+\.\d+\.\d+\.\d+|^\[\d+-\d+-\d+ \d+:\d+:\d+\]|^\d+:\d+:\d+.\d+|^\d+-\d+-\d+T\d+:\d+:\d+.\d+\+\d+:\d+|^\d{4}\\\d{2}\\\d{2} \d{2}:\d{2}:\d{2} \[|^\d+-\d+-\d+ \d+:\d+:\d+|^\d+:\d+:\d+|^\[\d+-[[:alpha:]]+-\d+ \d+:\d+:\d+\] '
    negate: true
    match: after

```

### But when I execute debug, it has a collection log

```auto
sh-4.2$ sudo filebeat -e -d "*"
2019-10-29T07:38:10.018Z INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-10-29T07:38:10.018Z DEBUG [beat] instance/beat.go:495 Beat metadata path: /var/lib/filebeat/meta.json
2019-10-29T07:38:10.019Z INFO instance/beat.go:475 Beat UUID: 7a4c3a40-9940-46f4-a9f7-33552ad11ae5
2019-10-29T07:38:10.019Z INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.1
2019-10-29T07:38:10.019Z DEBUG [beat] instance/beat.go:230 Initializing output plugins
2019-10-29T07:38:10.019Z DEBUG [processors] processors/processor.go:49 Processors:
2019-10-29T07:38:10.019Z INFO pipeline/module.go:76 Beat name: ip-10-10-12-206
2019-10-29T07:38:10.019Z INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2019-10-29T07:38:10.019Z INFO instance/beat.go:301 filebeat start running.
2019-10-29T07:38:10.020Z DEBUG [registrar] registrar/registrar.go:88 Registry file set to: /var/lib/filebeat/registry
2019-10-29T07:38:10.020Z INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry
2019-10-29T07:38:10.020Z INFO registrar/registrar.go:119 States Loaded from registrar: 6
2019-10-29T07:38:10.020Z WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If youhave already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-10-29T07:38:10.020Z INFO crawler/crawler.go:48 Loading Prospectors: 1
2019-10-29T07:38:10.020Z WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
2019-10-29T07:38:10.020Z DEBUG [processors] processors/processor.go:49 Processors:
2019-10-29T07:38:10.020Z DEBUG [registrar] registrar/registrar.go:150 Starting Registrar
2019-10-29T07:38:10.020Z DEBUG [prospector] log/config.go:178 recursive glob enabled
2019-10-29T07:38:10.020Z DEBUG [prospector] log/config.go:186 "/var/applog/**/*.log" expanded to []string{"/var/applog/*.log", "/var/applog/*/*.log", "/var/applog/*/*/*.log", "/var/applog/*/*/*/*.log", "/var/applog/*/*/*/*/*.log", "/var/applog/*/*/*/*/*/*.log", "/var/applog/*/*/*/*/*/*/*.log", "/var/applog/*/*/*/*/*/*/*/*.log", "/var/applog/*/*/*/*/*/*/*/*/*.log"}
2019-10-29T07:38:10.020Z DEBUG [prospector] log/config.go:186 "/var/www/webapp/logs/**/*.log" expanded to []string{"/var/www/webapp/logs/*.log", "/var/www/webapp/logs/*/*.log", "/var/www/webapp/logs/*/*/*.log", "/var/www/webapp/logs/*/*/*/*.log", "/var/www/webapp/logs/*/*/*/*/*.log", "/var/www/webapp/logs/*/*/*/*/*/*.log", "/var/www/webapp/logs/*/*/*/*/*/*/*.log", "/var/www/webapp/logs/*/*/*/*/*/*/*/*.log", "/var/www/webapp/logs/*/*/*/*/*/*/*/*/*.log"}
2019-10-29T07:38:10.021Z DEBUG [prospector] log/prospector.go:120 exclude_files: []. Number of stats: 6
2019-10-29T07:38:10.021Z DEBUG [prospector] file/state.go:82 New state added for /var/www/webapp/logs/A/B-FILES/B-FILES.log

```

> Why does debug start collecting when it doesn't work for a long time， Debug or restart and it will work properly

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 29, 2019, 7:36am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/2 "2019-10-29T07:36:09Z")

</div>

Could you please share the debug logs of Filebeat (`filebeat -e -d "*"`)?

---

<div class="post-metadata">

**Author:** ![Tom\_Chen1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_chen1/32/56752_2.png) [@Tom\_Chen1](https://discuss.elastic.co/u/Tom_Chen1)\
**Post date:** [October 29, 2019, 7:46am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/3 "2019-10-29T07:46:52Z")

</div>

Okay,

---

<div class="post-metadata">

**Author:** ![Tom\_Chen1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_chen1/32/56752_2.png) [@Tom\_Chen1](https://discuss.elastic.co/u/Tom_Chen1)\
**Post date:** [October 29, 2019, 8:07am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/4 "2019-10-29T08:07:23Z")

</div>

**The Filebeat daemon's log still don't show collect log action**

```auto
2019-10-29T07:37:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2560,"time":2569},"total":{"ticks":837
0,"time":8384,"value":8370},"user":{"ticks":5810,"time":5815}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25500009}},"memstats":{"gc_next":4851648,"memory_alloc":4239848,"memory
_total":1689268144}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0.02,"norm":{"1":0,"15":0,"5":0.01}}}}}}
2019-10-29T07:38:08.890Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2570,"time":2573},"total":{"ticks":839
0,"time":8394,"value":8390},"user":{"ticks":5820,"time":5821}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25530009}},"memstats":{"gc_next":5807264,"memory_alloc":4109248,"memory
_total":1691276976}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0.02,"norm":{"1":0,"15":0,"5":0.01}}}}}}
2019-10-29T07:38:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2570,"time":2573},"total":{"ticks":840
0,"time":8405,"value":8400},"user":{"ticks":5830,"time":5832}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25560009}},"memstats":{"gc_next":5605808,"memory_alloc":3406160,"memory
_total":1693288704}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0,"15":0,"5":0.02,"norm":{"1":0,"15":0,"5":0.01}}}}}}
2019-10-29T07:39:08.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2570,"time":2578},"total":{"ticks":840
0,"time":8415,"value":8400},"user":{"ticks":5830,"time":5837}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25590009}},"memstats":{"gc_next":5960912,"memory_alloc":2978296,"memory
_total":1695300032}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0.08,"15":0.01,"5":0.03,"norm":{"1":0.04,"15":0.005,"5":0.015}}}}}}
2019-10-29T07:39:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2580,"time":2582},"total":{"ticks":842
0,"time":8423,"value":8420},"user":{"ticks":5840,"time":5841}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25620009}},"memstats":{"gc_next":5960912,"memory_alloc":4988640,"memory
_total":1697310376}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0.05,"15":0,"5":0.03,"norm":{"1":0.025,"15":0,"5":0.015}}}}}}
2019-10-29T07:40:08.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2580,"time":2586},"total":{"ticks":842
0,"time":8433,"value":8420},"user":{"ticks":5840,"time":5847}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25650009}},"memstats":{"gc_next":4894432,"memory_alloc":4261640,"memory
_total":1699319256}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0.25,"15":0.02,"5":0.07,"norm":{"1":0.125,"15":0.01,"5":0.035}}}}}}
2019-10-29T07:40:38.891Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2590,"time":2591},"total":{"ticks":844
0,"time":8443,"value":8440},"user":{"ticks":5850,"time":5852}},"info":{"ephemeral_id":"e0bd341a-6d22-4d36-937a-ec477851f9f3","uptime":{"ms":25680009}},"memstats":{"gc_next":4890496,"memory_alloc":4265144,"memory
_total":1701334744}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":1,"events":{"active":459}}},"registrar
":{"states":{"current":1}},"system":{"load":{"1":0.15,"15":0.01,"5":0.06,"norm":{"1":0.075,"15":0.005,"5":0.03}}}}}}

```

---

<div class="post-metadata">

**Author:** ![Tom\_Chen1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_chen1/32/56752_2.png) [@Tom\_Chen1](https://discuss.elastic.co/u/Tom_Chen1)\
**Post date:** [October 29, 2019, 10:37am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/5 "2019-10-29T10:37:23Z")

</div>

## We have a command that dynamically updates the configuration file, probably because we changed the format of the configuration file

```auto
filebeat.prospectors:
- input_type: log
  paths:
    - /var/applog/**/*.log
    - /var/www/webapp/logs/*/*.log
    - /var/www/webapp/logs/**/*.log

```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 29, 2019, 10:51am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/6 "2019-10-29T10:51:06Z")

</div>

Could you please share the debug logs when starting Filebeat. I would like to see if Filebeat is able to find the files in your configuration.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2019, 10:51am UTC](https://discuss.elastic.co/t/filebeat6-2-1-logs-are-not-collected-intermittently/205607/7 "2019-11-26T10:51:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.