# How to "exclude\_lines but make these exceptions"? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 10, 2016, 8:34pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917 "2016-04-10T20:34:03Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![erikstephens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erikstephens/32/5430_2.png) [@erikstephens](https://discuss.elastic.co/u/erikstephens)\ **Post date:** [April 10, 2016, 8:34pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/1 "2016-04-10T20:34:03Z")
I'm shipping syslog with custom template to log the severity. I'm using `exclude_lines` to not bother shipping anything with severity greater than severity info. I'd like to make an exception for a handful of processes that are logging interesting msgs at debug level. A quick test shows I can have multiple prospectors process the same file. Is this behavior expected? I'm worried that I may be abusing it some how and this may change in the future. Details of my quick test are below if helps provide more context. Thanks! ```auto $ cat filebeat.yml output: console: pretty: true filebeat: prospectors: - paths: - /tmp/test.log include_lines: - 'KEEP' - paths: - /tmp/test.log exclude_lines: - 'SEV:[6-9]' $ cat test.log NOISE SEV:6 KEEP SEV:6 NOISE SEV:7 SIGNAL SEV:4 $ rm -f .filebeat; filebeat { "@timestamp": "2016-04-10T20:29:58.958Z", "beat": { "hostname": "home", "name": "home" }, "count": 1, "fields": null, "input_type": "log", "message": "SIGNAL SEV:4", "offset": 35, "source": "/tmp/test.log", "type": "log" } { "@timestamp": "2016-04-10T20:29:58.959Z", "beat": { "hostname": "home", "name": "home" }, "count": 1, "fields": null, "input_type": "log", "message": "KEEP SEV:6", "offset": 12, "source": "/tmp/test.log", "type": "log" } ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 11, 2016, 6:16am UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/2 "2016-04-11T06:16:53Z")
I would strongly recommend not to do that as it will mess with the registry file. Currently the registry file uses the path as identifier so there will be two processes modifying the same state data. I assume this will lead to some wired behaviour. The current solution would be to start two instance of filebeat and configuring two different locations for the registry file. Some refactoring in the registry file is hopefully happening in the near future. Part of this refactoring could make the above usage possible, in case the identifier of a harvested file is unique so the same file could be harvested by different prospectors without having conflicts in the registry file. See here for some more details: [https://github.com/elastic/beats/issues/1022#issuecomment-206737447](https://github.com/elastic/beats/issues/1022#issuecomment-206737447) ---
**Author:** ![erikstephens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erikstephens/32/5430_2.png) [@erikstephens](https://discuss.elastic.co/u/erikstephens)\ **Post date:** [April 11, 2016, 3:08pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/3 "2016-04-11T15:08:18Z")
Bummer. It's probably too late, but I wish `include_lines` behaved more as an exception list for `exclude_lines` - could support the current behavior with `exclude_lines: ['.']`. Thanks for the reply, though. Saved me a nasty surprise later. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 12, 2016, 6:29am UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/4 "2016-04-12T06:29:44Z")
Here are some more details on the exclude\_lines implementation: [https://github.com/elastic/beats/pull/430](https://github.com/elastic/beats/pull/430) Not sure i understand what you mean with `['.']` ---
**Author:** ![erikstephens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erikstephens/32/5430_2.png) [@erikstephens](https://discuss.elastic.co/u/erikstephens)\ **Post date:** [April 12, 2016, 3:38pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/5 "2016-04-12T15:38:14Z")
If willing to entertain the idea of making include\_lines behave more like an exception list for exclude\_lines, then users could use `exclude_lines: ['.']` along with their current include\_lines to achieve the current behavior. I'm throwing that out there as an option to avoid adding another option (eg `exclude_lines_exceptions`). It's odd to me since before include\_lines was available, all lines were already included. Someone interested in using include\_lines is really interested in excluding. Then the question becomes how does include\_lines interact with exclude\_lines. I wish it was more like: ```auto if len(h.ExcludeLinesRegexp) > 0 { if MatchAnyRegexps(h.ExcludeLinesRegexp, line) { if len(h.IncludeLinesRegexp) > 0 { if !MatchAnyRegexps(h.IncludeLinesRegexp, line) { logp.Debug("harvester", "Drop line as it matches an exclude pattern and does not match any of the include patterns %s", line) return false } } else { logp.Debug("harvester", "Drop line as it matches an exclude pattern and no include patterns defined %s", line) return false } } } return true ``` I can't find any other discussion about this, so I'm probably just being weird - feel free to ignore 🙂 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 13, 2016, 7:54am UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/6 "2016-04-13T07:54:54Z")
As @monica is working on this it probably makes sense that she chimes in here. ---
**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\ **Post date:** [April 13, 2016, 9:50am UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/7 "2016-04-13T09:50:58Z")
The way is implemented at the moment is that `include_lines` is called before `exclude_lines`. The idea was to be able to include Apache logs but exclude the debug messages coming from Apache by setting: ` include_lines: ['apache'] exclude_lines: ['^DBG'] ` ---
**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\ **Post date:** [April 13, 2016, 10:03am UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/8 "2016-04-13T10:03:10Z")
We are currently working on a generic solution for filtering that would be part of libbeat and available for all the Beats where you can easily choose the fields to export or the events to drop based on a condition. Your use case is interesting and we will consider it when implementing generic filtering. ---
**Author:** ![erikstephens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erikstephens/32/5430_2.png) [@erikstephens](https://discuss.elastic.co/u/erikstephens)\ **Post date:** [April 13, 2016, 2:17pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/9 "2016-04-13T14:17:50Z")
Where is that conversation happening? I'm subscribed to [this issue](https://github.com/elastic/beats/issues/451) but there hasn't been much activity lately. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:53pm UTC](https://discuss.elastic.co/t/how-to-exclude-lines-but-make-these-exceptions/46917/10 "2017-07-05T21:53:20Z")