# Harvesting same file with different prospector config **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 15, 2017, 11:52am UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546 "2017-06-15T11:52:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![monitoring\_it](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@monitoring\_it](https://discuss.elastic.co/u/monitoring_it)\ **Post date:** [June 15, 2017, 11:52am UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/1 "2017-06-15T11:52:44Z")
Does anyone see any issue with harvesting same file from different filebeat prospectors each having its own custom fields and set of include exclude regexs? My usecase is, i want filebeat to parse a single file for certain set of include and exclude patterns and set different type field value. Then using when condition in filebeat config send the parsed data to different indexes in elasticsearch. Filebeat seems to do that pretty neatly but does anyone sees any issue with this approach? **Version - 5.1.1-1** ```auto filebeat: prospectors: - paths: - /var/log/test.log input_type: log document_type: alert ignore_older: 24h exclude_lines: ['^DBG'] include_lines: ['^ERROR'] scan_frequency: 10s backoff: 1s max_backoff: 10s fields: sev: "MAJOR" label: "ERROR" backoff_factor: 2 force_close_files: false fields_under_root: false close_older: 2h - paths: - /var/log/test.log input_type: log document_type: alert ignore_older: 24h exclude_lines: ['java\.lang\.IllegalArgumentException: Document base'] include_lines: ['ERROR LogMananger\.repositorySelector was null'] scan_frequency: 10s backoff: 1s max_backoff: 10s fields: sev: "MINOR" label: "repo-issue" backoff_factor: 2 force_close_files: false fields_under_root: false close_older: 2h - paths: - /var/log/test.log input_type: log document_type: alert ignore_older: 24h exclude_lines: ['java\.lang\.IllegalArgumentException: Document base']** include_lines: ['SEVERE: Servlet\.service','Stopping service Catalina']** scan_frequency: 10s backoff: 1s max_backoff: 10s fields: sev: "MINOR" label: "Servlet_Failure" backoff_factor: 2 force_close_files: false fields_under_root: false close_older: 2h output: elasticsearch: hosts: ['https://xxxxxxxxxx:443'] index: filebeat-%{+yyyy.MM.dd} indices: - index: "alert-%{+yyyy.MM.dd}" when.contains: type: "alert" name: grafana-mon-GrafanaApp-15FIOD3L34PR ``` ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [June 15, 2017, 2:28pm UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/2 "2017-06-15T14:28:02Z")
Defining the same file in multiple prospectors will cause problems due to how the read offset is persisted to disk. And in later versions of Filebeat I suspect it will yield an error on startup. Using Logstash to annotate the events would make sense. Additionally I could see this being a feature of Beats whereby processors could be used to conditionally add tags and fields. Like: ```auto processors: - add_fields: when.regexp.message: `repositorySelector` fields: label: "repo-issue" ``` ---
**Author:** ![monitoring\_it](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@monitoring\_it](https://discuss.elastic.co/u/monitoring_it)\ **Post date:** [June 15, 2017, 2:50pm UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/3 "2017-06-15T14:50:37Z")
So far i havent seen any issue in the test environment. I do see that in registry file there is one inode used for test.log file and offset keeps changing when i echo a messages in the log file. Can you shed some light on why do u think the way offsets are created and stored will cause an issue? What if i keep other prospector config same ( that deals with closing/scanning files) but only change fields and regexs? Also is processor feature available today within filebeat to be used. If yes, is there a not operator available in when regex condition so that i can say add fields only if message matches ABC and also doesnt match XYZ ( similar to include\_line and exclude\_line feature) ---
**Author:** ![monitoring\_it](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@monitoring\_it](https://discuss.elastic.co/u/monitoring_it)\ **Post date:** [June 15, 2017, 2:58pm UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/4 "2017-06-15T14:58:22Z")
I think i found the link on processors. It definitely looks like that it handles my usecase. [https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-processors.html](https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-processors.html). This feature looks like is available from v5.4 onwards. I am going to try it. Thanks for pointing me to right direction. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [June 15, 2017, 3:00pm UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/5 "2017-06-15T15:00:48Z")
[Processors](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-processors.html) are available today, but there is no "add\_fields" processor. There will be 3 separate readers. Each will be persisting its read offset to the registry. But only one entry will be stored in the registry file. When you restart Filebeat all three readers will resume from the same persisted offset, but that offset is possibly wrong for two of the three readers. So you either re-read some lines or your miss some lines because they were skipped over. ---
**Author:** ![monitoring\_it](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@monitoring\_it](https://discuss.elastic.co/u/monitoring_it)\ **Post date:** [June 15, 2017, 3:13pm UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/6 "2017-06-15T15:13:09Z")
Thanks for the explanation on readers. Makes sense! Do you think it makes sense to add\_fields processor? Any timeline when it can be expected? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [June 16, 2017, 1:54am UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/7 "2017-06-16T01:54:08Z")
I think `add_fields` would be useful in some situations. Do you want to open an enhancement request on Github for it? ---
**Author:** ![monitoring\_it](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@monitoring\_it](https://discuss.elastic.co/u/monitoring_it)\ **Post date:** [June 16, 2017, 1:59am UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/8 "2017-06-16T01:59:05Z")
Will definitely do. Thx!! I was able to use processors for my usecase although a little clumsy. Having option to add fields will greatly help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 14, 2017, 1:59am UTC](https://discuss.elastic.co/t/harvesting-same-file-with-different-prospector-config/89546/9 "2017-07-14T01:59:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.