# Process more than 3000 lines per record in filebeat

**URL:** <https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 14, 2018, 1:58pm UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752 "2018-05-14T13:58:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kerensu](https://avatars.discourse-cdn.com/v4/letter/k/c37758/32.png) [@kerensu](https://discuss.elastic.co/u/kerensu)\
**Post date:** [May 14, 2018, 1:58pm UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752/1 "2018-05-14T13:58:45Z")

</div>

```
  When using the multiline.max_lines and multiline.time_out file beat does no pick any record.
  Since the records are bigger than default I must set multiline.max_lines, correct? Please advice how to do that correctly.

```

Here is the prospector:  
- type: log  
enabled: true  
paths:  
- /home/vpwrk1/ElasticDataForTest/test\*.log  
fields:  
type: connectoromsmngsrv  
fields\_under\_root: true  
exclude\_lines: ["-------------------"]  
multiline.pattern: '^[[:graph:]]{3}\s[a-zA-Z]+\s\d+[[:graph:]]\s\d{4}\s\d+[[:graph:]]\d+[[:graph:]]\d+\s[A-Z]+\s[A-Z]+\s[[:graph:]]{3}'  
multiline.negate: true  
multiline.match: after  
multiline.max\_lines: 4000  
multiline.timeout: 4

thanks

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 14, 2018, 4:06pm UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752/2 "2018-05-14T16:06:28Z")

</div>

Yes, for records with more than 500 lines you need to set `max_lines`. Can you see any related error in filebeat logs?

---

<div class="post-metadata">

**Author:** ![kerensu](https://avatars.discourse-cdn.com/v4/letter/k/c37758/32.png) [@kerensu](https://discuss.elastic.co/u/kerensu)\
**Post date:** [May 16, 2018, 2:48pm UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752/3 "2018-05-16T14:48:00Z")

</div>

Today when I set max\_lines to 3500 ( record size is ~ 3350) I got this in the log:  
2018-05-16T16:31:33+03:00 DBG Drop line as it does match one of the exclude patterns\*\*\* January 25, 2018 8:03:40 PM PHT \*\*\*  
[1] Executing 'GetCustomerList' contract  
However when I set max\_lines to 2500 the SAME record ( dropped above) is passed to logstash, and filebeat has no issue with the date. In this case of course logstash receives a cut to 2500 lines record. How can I set max\_lines to 3500 and receive the whole record in logstash?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [May 17, 2018, 8:21am UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752/4 "2018-05-17T08:21:22Z")

</div>

It's important to mention that exclude\_lines is applied after the multiline. So it seems if the complete multiline is read in, it contains your exclude pattern and is dropped. Try to remove or modify your exclude pattern.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2018, 8:21am UTC](https://discuss.elastic.co/t/process-more-than-3000-lines-per-record-in-filebeat/131752/5 "2018-06-14T08:21:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.