# Filebeat collects logs prior to deployment of ELK

**URL:** <https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 15, 2018, 4:20am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446 "2018-08-15T04:20:21Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 15, 2018, 4:20am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/1 "2018-08-15T04:20:21Z")

</div>

Hi  
I'm using ELk-6.3.1 I've only recently started using Elk, and I output some of the cluster logs to logstash through filebeat, but I can only deliver the latest ones, and before I deploy ELk, will some logs also want to collect analysis filebeat to be implemented?  
Please advise. Thank you.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [August 15, 2018, 4:36am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/2 "2018-08-15T04:36:20Z")

</div>

Hi ss,

Are the logs written to a file? If so, you can process them whenever you want as long as they are not rotated.

For other forms of input such as TCP/UDP and so on, you will need to build either an HA in Logstash or some other redundancy mechanism.

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 15, 2018, 5:53am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/3 "2018-08-15T05:53:13Z")

</div>

Hi  
first of all, thank you very much for replying to me.  
I know very well that ELK can collect the generated logs in real time. The problem is that some of the logs are generated by some programs running before the deployment of ELK, and the requirement is that you can collect these logs as well. Do you have any good ways?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 15, 2018, 10:46am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/4 "2018-08-15T10:46:35Z")

</div>

I am not sure I understood correctly what your requirements are. I assume you want to collect logs from files which were written previously by a program, before Filebeat is started. Please, correct me if it's not the case.

Filebeat is capable of reading from files which are not written to anymore. Filebeat starts to read it, when finished it closes the FD. It tries to read it every time `scan_frequency` has elasped, but if the file hasn't change since Filebeat last encountered it, FB does not read it again. Is it possible Filebeat already read that file before, so it does not pick up it again?

Could you please share your config formatted using `</>`? Also, please attach the debug logs of Filebeat (`./filebeat -e -d "*"`).

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 16, 2018, 2:31am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/5 "2018-08-16T02:31:34Z")

</div>

Hello  
First of all, thank you very much for being able to reply to my questions, as you said, I need to use filebeat to read the history log, that is, not to write the file; I also see in the official documentation that the ignore\_older parameter is described as having access to last week's file and the most recent file, but I configured it not to take effect, and I don't understand the use of this parameter! My understanding is that I start filebeat. by setting this parameter, I can get the log a week ago. Is that right? Or do you have any other solutions?  
Please advise. Thank you.

The following is the configuration file for filebeat:

```
filebet: 

- type: log 
  tail_files: false
  ignore_older: 48h 
  enabled: true
paths:
  - /var/log/messages*
fields:
  document_type: message_log

#============================= Filebeat modules ===============================

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml

  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

#-------------------------- Logstash output ------------------------------
output.logstash:
  hosts: ["172.16.0.5:5044"]
```

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 16, 2018, 2:37am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/6 "2018-08-16T02:37:57Z")

</div>

This is the filebeat debug log:

[root@elk\_test1 /etc/filebeat]  
@ 10:28:27 @# /usr/share/filebeat/bin/filebeat -e -d "_"  
2018-08-16T10:29:50.568+0800 INFO instance/beat.go:492 Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]  
2018-08-16T10:29:50.568+0800 DEBUG [beat] instance/beat.go:519 Beat metadata path: /usr/share/filebeat/bin/data/meta.json  
2018-08-16T10:29:50.568+0800 INFO instance/beat.go:499 Beat UUID: 068e423e-9cf8-4ed5-a1c3-567feca264fb  
2018-08-16T10:29:50.568+0800 INFO [beat] instance/beat.go:716 Beat info {"system\_info": {"beat": {"path": {"config": "/usr/share/filebeat/bin", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs"}, "type": "filebeat", "uuid": "068e423e-9cf8-4ed5-a1c3-567feca264fb"}}}  
2018-08-16T10:29:50.568+0800 INFO [beat] instance/beat.go:725 Build info {"system\_info": {"build": {"commit": "ed42bb85e72ae58cc09748dc1825159713e0ffd4", "libbeat": "6.3.1", "time": "2018-06-29T21:09:35.000Z", "version": "6.3.1"}}}  
2018-08-16T10:29:50.568+0800 INFO [beat] instance/beat.go:728 Go runtime info {"system\_info": {"go": {"os":"linux","arch":"amd64","max\_procs":1,"version":"go1.9.4"}}}  
2018-08-16T10:29:50.569+0800 INFO [beat] instance/beat.go:732 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2018-08-15T09:28:14+08:00","containerized":true,"hostname":"elk\_test1","ips":["127.0.0.1/8","::1/128","172.16.0.2/32","fe80::e6c5:4ad9:9c0c:5a0f/64"],"kernel\_version":"3.10.0-693.el7.x86\_64","mac\_addresses":["00:0c:29:05:46:20"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":4,"patch":1708,"codename":"Core"},"timezone":"CST","timezone\_offset\_sec":28800,"id":"2e58c05f4aa24e779395b4f65903af39"}}}  
2018-08-16T10:29:50.569+0800 INFO [beat] instance/beat.go:761 Process info {"system\_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac\_override","dac\_read\_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux\_immutable","net\_bind\_service","net\_broadcast","net\_admin","net\_raw","ipc\_lock","ipc\_owner","sys\_module","sys\_rawio","sys\_chroot","sys\_ptrace","sys\_pacct","sys\_admin","sys\_boot","sys\_nice","sys\_resource","sys\_time","sys\_tty\_config","mknod","lease","audit\_write","audit\_control","setfcap","mac\_override","mac\_admin","syslog","wake\_alarm","block\_suspend"],"effective":["chown","dac\_override","dac\_read\_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux\_immutable","net\_bind\_service","net\_broadcast","net\_admin","net\_raw","ipc\_lock","ipc\_owner","sys\_module","sys\_rawio","sys\_chroot","sys\_ptrace","sys\_pacct","sys\_admin","sys\_boot","sys\_nice","sys\_resource","sys\_time","sys\_tty\_config","mknod","lease","audit\_write","audit\_control","setfcap","mac\_override","mac\_admin","syslog","wake\_alarm","block\_suspend"],"bounding":["chown","dac\_override","dac\_read\_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux\_immutable","net\_bind\_service","net\_broadcast","net\_admin","net\_raw","ipc\_lock","ipc\_owner","sys\_module","sys\_rawio","sys\_chroot","sys\_ptrace","sys\_pacct","sys\_admin","sys\_boot","sys\_nice","sys\_resource","sys\_time","sys\_tty\_config","mknod","lease","audit\_write","audit\_control","setfcap","mac\_override","mac\_admin","syslog","wake\_alarm","block\_suspend"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 4479, "ppid": 1168, "seccomp": {"mode":"disabled"}, "start\_time": "2018-08-16T10:29:49.810+0800"}}}  
2018-08-16T10:29:50.569+0800 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.1  
2018-08-16T10:29:50.569+0800 DEBUG [beat] instance/beat.go:242 Initializing output plugins  
2018-08-16T10:29:50.569+0800 DEBUG [processors] processors/processor.go:49 Processors:  
2018-08-16T10:29:50.570+0800 DEBUG [publish] pipeline/consumer.go:120 start pipeline event consumer  
2018-08-16T10:29:50.570+0800 INFO pipeline/module.go:81 Beat name: elk\_test1  
2018-08-16T10:29:50.570+0800 ERROR fileset/modules.go:101 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module  
2018-08-16T10:29:50.570+0800 INFO instance/beat.go:315 filebeat start running.  
2018-08-16T10:29:50.570+0800 DEBUG [registrar] registrar/registrar.go:96 Registry file set to: /usr/share/filebeat/bin/data/registry  
2018-08-16T10:29:50.570+0800 INFO registrar/registrar.go:116 Loading registrar data from /usr/share/filebeat/bin/data/registry  
2018-08-16T10:29:50.570+0800 INFO registrar/registrar.go:127 States Loaded from registrar: 0  
2018-08-16T10:29:50.570+0800 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.  
2018-08-16T10:29:50.570+0800 INFO crawler/crawler.go:48 Loading Inputs: 0  
2018-08-16T10:29:50.570+0800 DEBUG [cfgfile] cfgfile/reload.go:90 Checking module configs from: /usr/share/filebeat/bin/modules.d/_.yml  
2018-08-16T10:29:50.570+0800 DEBUG [cfgfile] cfgfile/reload.go:104 Number of module configs found: 0  
2018-08-16T10:29:50.570+0800 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 0  
2018-08-16T10:29:50.570+0800 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s  
2018-08-16T10:29:50.570+0800 DEBUG [registrar] registrar/registrar.go:158 Starting Registrar  
2018-08-16T10:29:50.570+0800 INFO cfgfile/reload.go:122 Config reloader started

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 16, 2018, 2:38am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/7 "2018-08-16T02:38:29Z")

</div>

2018-08-16T10:29:50.570+0800 DEBUG [cfgfile] cfgfile/reload.go:146 Scan for new config files  
2018-08-16T10:29:50.570+0800 DEBUG [cfgfile] cfgfile/reload.go:165 Number of module configs found: 0  
2018-08-16T10:29:50.570+0800 INFO cfgfile/reload.go:214 Loading of config files completed.  
2018-08-16T10:30:20.574+0800 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":2}},"total":{"ticks":0,"time":{"ms":12},"value":0},"user":{"ticks":0,"time":{"ms":10}}},"info":{"ephemeral\_id":"72af64bf-4c44-44de-9b8c-20172c425a20","uptime":{"ms":30006}},"memstats":{"gc\_next":4473924,"memory\_alloc":3087656,"memory\_total":3087656,"rss":11964416}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.05,"5":0.01}}}}}}  
^C2018-08-16T10:30:23.679+0800 DEBUG [service] service/service.go:34 Received sigterm/sigint, stopping  
2018-08-16T10:30:23.679+0800 INFO beater/filebeat.go:420 Stopping filebeat  
2018-08-16T10:30:23.679+0800 INFO crawler/crawler.go:109 Stopping Crawler  
2018-08-16T10:30:23.679+0800 INFO crawler/crawler.go:119 Stopping 0 inputs  
2018-08-16T10:30:23.679+0800 INFO cfgfile/reload.go:217 Dynamic config reloader stopped  
2018-08-16T10:30:23.679+0800 INFO crawler/crawler.go:135 Crawler stopped  
2018-08-16T10:30:23.679+0800 INFO registrar/registrar.go:247 Stopping Registrar  
2018-08-16T10:30:23.679+0800 INFO registrar/registrar.go:173 Ending Registrar  
2018-08-16T10:30:23.679+0800 DEBUG [registrar] registrar/registrar.go:291 Write registry file: /usr/share/filebeat/bin/data/registry

As you said, you can read history logs that are not being written, and please provide a solution that would be highly appreciated for your help

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 16, 2018, 6:59am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/8 "2018-08-16T06:59:24Z")

</div>

Your configuration seems incorrect.

```auto
filebeat.inputs: 

- type: log 
  tail_files: false
  ignore_older: 48h 
  enabled: true
  paths:
    - /var/log/messages*
  fields:
    document_type: message_log

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

#-------------------------- Logstash output ------------------------------
output.logstash:
  hosts: ["172.16.0.5:5044"]

```

`fields` part of your configuration is not straightforward. Do you want to add the field you specified to every event produced by Filebeat or only to events coming from the log input you added?

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 16, 2018, 7:20am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/9 "2018-08-16T07:20:22Z")

</div>

Hi  
Thanks for pointing out my drawbacks, I'm still in the test phase, so some features aren't configured in too much detail, just to read new logs if you want to read logs that are not written. And it was generated before the filebeat deployment, can you point out how to configure it?  
Please advise. Thank you.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 16, 2018, 1:44pm UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/10 "2018-08-16T13:44:27Z")

</div>

I am sorry, but I might not understood what you asked previously. To clear things up, do you want to skip old files? And only read files which are written after Filebeat is started?

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 17, 2018, 1:34am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/11 "2018-08-17T01:34:08Z")

</div>

Sorry, Maybe I'm not talking about it in detail. Now let's say a scenario like this: running the application service on some servers in a production environment will generate a lot of logs, now deploy filebeat.To make it easier to view the log. How does the requirement collect the logs generated before the deployment of filebeat?

Thank you for any advice and help

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 17, 2018, 9:42am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/12 "2018-08-17T09:42:37Z")

</div>

Ok, I think I got it now. The contradiction which confused me is you have `ignore_older: 48h`. This option tells Filebeat to ignore every input file which is older than 2 days. If that's not what you want, delete this from the config.

If Filebeat still does not pick up your files, it means Filebeat has already encountered those files and successfully forwarded them to the configured output. If you are sure the logs haven't arrived, you could delete `data/registry`. However, it could lead to log duplication if the files you cannot find right now, are not really missing. So be careful.

---

<div class="post-metadata">

**Author:** ![ss.z](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@ss.z](https://discuss.elastic.co/u/ss.z)\
**Post date:** [August 21, 2018, 2:25am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/13 "2018-08-21T02:25:55Z")

</div>

OK, thank you for your answers to my questions and suggestions, I will take them!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2018, 2:26am UTC](https://discuss.elastic.co/t/filebeat-collects-logs-prior-to-deployment-of-elk/144446/14 "2018-09-18T02:26:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.