# Not all the files picked at the same time

**URL:** <https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 29, 2018, 11:40am UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956 "2018-06-29T11:40:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akilen\_Pandian](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Akilen\_Pandian](https://discuss.elastic.co/u/Akilen_Pandian)\
**Post date:** [June 29, 2018, 11:40am UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/1 "2018-06-29T11:40:21Z")

</div>

Hi There

I am new to filebeat. I am setting up logstash server to monitor log files from different process. Process run at a fix time and output is stored in a log file.

This is my config  
`  
filebeat:  
prospectors:  
-  
paths:  
- /var/log/auth.log  
input\_type: log  
document\_type: syslog  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22609-6240/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22548-6238/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22315-6237/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22308-6239/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22290-6236/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/\_logs\_22289-6235/out.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage  
-  
paths:  
- /home/deltion/kinaxia/stage/_/error.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
backoff: 600s  
backoff\_factor: 1  
document\_type: kinaxia\_stage\_error  
-  
paths:  
- /home/deltion/CNOAPI/stage/_.log  
multiline.pattern: '^([0-9]{2}:[0-9]{2}:[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
document\_type: automata\_stage\_log  
-  
paths:  
- /home/deltion/CNOAPI/test/\*.log  
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'  
multiline.negate: true  
multiline.match: after  
input\_type: log  
document\_type: automata\_test\_log

registry\_file: /var/lib/filebeat/registry

output:  
logstash:  
hosts: ["127.0.0.1:5044"]  
bulk\_max\_size: 1024  
tls:  
certificate\_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]  
file:  
enabled: true  
path: /home/deltion/filebeat\_log  
filename: filebeat  
rotate\_every\_kb: 10485760  
number\_of\_files: 10

shipper:

logging:  
to\_files: true  
to\_syslog: false  
files:  
rotateeverybytes: 10485760 # = 10MB  
path: /var/log/mybeat  
name: mybeat.log  
keepfiles: 7  
level: info  
`

I don't see the out.log file from all the folders at set time.

Can you please tell me what i am doing wrong?

Thanks for your help in advance

Akilen

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [June 29, 2018, 12:55pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/2 "2018-06-29T12:55:04Z")

</div>

> Process run at a fix time and output is stored in a log file.

Do you mean that you manually trigger starting of Filebeat and kill it after?  
I am not sure I understand the "at set time" in your case.

---

<div class="post-metadata">

**Author:** ![Akilen\_Pandian](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Akilen\_Pandian](https://discuss.elastic.co/u/Akilen_Pandian)\
**Post date:** [June 29, 2018, 2:13pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/3 "2018-06-29T14:13:56Z")

</div>

for example  
/home/deltion/kinaxia/stage/\_logs\_22609-6240/out.log  
and  
/home/deltion/kinaxia/stage/\_logs\_22548-6238/out.log

need to be picked every 10min, but only one of the file will be processed

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [June 29, 2018, 3:43pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/4 "2018-06-29T15:43:10Z")

</div>

Hello @Akilen_Pandian

Is the "out.log" a complete new file every 10 minutes?

---

<div class="post-metadata">

**Author:** ![Akilen\_Pandian](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Akilen\_Pandian](https://discuss.elastic.co/u/Akilen_Pandian)\
**Post date:** [June 29, 2018, 4:28pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/5 "2018-06-29T16:28:47Z")

</div>

Yes it is

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [June 29, 2018, 4:48pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/6 "2018-06-29T16:48:38Z")

</div>

@Akilen_Pandian

I have a few suggestions that should help in your case, but I will have the following assumptions:

1. The out file is created every 10 minutes.
2. The out file get a new inode on every creation (Filebeat tracks inodes for new file disambiguation)

You can do the following.

You don't need to define a new input per log, you can instead use wildcards.

```auto
paths:
- /home/deltion/kinaxia/stage/*/out.log
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'
multiline.negate: true
multiline.match: after
input_type: log
document_type: kinaxia_stage

```

**Note:** I've removed the **backoff** and the **backoff\_factor** option, I think this is why you don't get all the logs. By removing them Filebeat will scan and detect new files as they happen.

I don't know the nature or the size of the logs you are currently monitoring, but it could be possible that the file is removed when Filebeat is still processing it or when there is back pressure on the outputs, this could lead to data loss.

So I think it would wise to think about the following:

1. Are you controlling that rotation?
2. Instead of replacing the file could we create another file next to the previous one? You could have a process to remove older after a few hours or days.

---

<div class="post-metadata">

**Author:** ![Akilen\_Pandian](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Akilen\_Pandian](https://discuss.elastic.co/u/Akilen_Pandian)\
**Post date:** [June 29, 2018, 7:44pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/7 "2018-06-29T19:44:50Z")

</div>

When I started I had the configuration as you suggested, but I had the same problem. So I changed will this fix my problem

To answer your question

1. No I don't have control over the log rotation.
2. I need to talk with my system team to see can they generate new log file every time it run

Thanks

---

<div class="post-metadata">

**Author:** ![Akilen\_Pandian](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Akilen\_Pandian](https://discuss.elastic.co/u/Akilen_Pandian)\
**Post date:** [June 29, 2018, 9:36pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/8 "2018-06-29T21:36:10Z")

</div>

Hi @pierhugues

Thanks for your help.

As you suspected the issue was because of the file getting replace when it is getting processed

I have asked the system admin the change the process

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2018, 9:39pm UTC](https://discuss.elastic.co/t/not-all-the-files-picked-at-the-same-time/137956/9 "2018-07-27T21:39:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.