# Difference in sending static vs "live feeding" logs to logstash via filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 3, 2023, 10:23am UTC](https://discuss.elastic.co/t/difference-in-sending-static-vs-live-feeding-logs-to-logstash-via-filebeat/346349 "2023-11-03T10:23:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 3, 2023, 10:23am UTC](https://discuss.elastic.co/t/difference-in-sending-static-vs-live-feeding-logs-to-logstash-via-filebeat/346349/1 "2023-11-03T10:23:51Z")
We have two different ELK servers which are used to analyse logs from our own application, one is in-house and the other is on the customers site. We have different pipelines for the application itself and then also for the various application services. Example we have application.log pipeline, then application.service1.log, application.service2.log and so on in the The in-house setup gathers the logs from a share after they are moved there by a script and are static in that they are no longer being changed or updated. Filebeat is configured to push the logs into logstash. On the customer site we have multiple application servers with filebeat running, pushing the the logs directly to logstash over the network to the ELK server. Some of the logs are constantly being amended and added to over time We have noticed that the logs are being indexed differently on the customer site. On the customer site there appears to be some "cross contamination" between the application.log and application.server1.log logs when we see log lines in the message field that shouldn´t be there. Our question is, should there be an difference between sending static logs vs a live feed to logstash and if so what filebeat input options (or elasticesearch output?) should be applied in this "live feed scenario" Apologies if some of the terminology is off its because I am new to ELK stack. Thanks in advance ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 3, 2023, 1:05pm UTC](https://discuss.elastic.co/t/difference-in-sending-static-vs-live-feeding-logs-to-logstash-via-filebeat/346349/2 "2023-11-03T13:05:02Z")
> [@randomnamegenerator](#): > > Our question is, should there be an difference between sending static logs vs a live feed to logstash and if so what filebeat input options (or elasticesearch output?) There is basically no difference, FIlebeat is line oriented, it will start reading the files and processing line by line, the only difference is that when file are constantly being written, they may be rotated and depending on the configuration you can have duplicate messags or miss some logs. > [@randomnamegenerator](#): > > On the customer site there appears to be some "cross contamination" between the application.log and application.server1.log logs when we see log lines in the message field that shouldn´t be there. It is not clear what you mean with that, can you provide more context? Are the logs on `application.log` different from the logs on `application.server1.log`? You need to provide more context about this issue like the `filebeat.yml`, the logstash configuration, what is your output and what is the expected output. ---
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 5, 2023, 7:50pm UTC](https://discuss.elastic.co/t/difference-in-sending-static-vs-live-feeding-logs-to-logstash-via-filebeat/346349/3 "2023-11-05T19:50:06Z")
Apologies, I am a newbie so my ELK terminology might be off. And also I made a typo, application.server1.log should be application.service1.log. I will try add context best as I can. The filebeat.yml is as below filebeat.registry.path: ./filebeat logging.to\_files: true logging.files: path: C:\Company\Data\Logs\Monitoring name: filebeat.log rotateeverybytes: 10485760 keepfiles: 2 logging.level: info filebeat.spool\_size: 102400 filebeat.idle\_timeout: 15s filebeat.inputs: - input\_type: log paths: - input\_type: log paths: output.logstash: hosts: ["ESServer:5044"] bulk\_max\_size: 8192 worker: 4 compression\_level: 0 pipelining: 5 We then have separate logstash filter configs for both Application.Server and Application.Service. The Application.Service.config is as below, its missing some additional groks for the sake of brevity. filter { if [log\_type] == "Application\_Service\_log\_files" { ``` fingerprint { source => ["Site", "message", "source", "log_date", "logger_name", "tool_name", "level"] concatenate_sources => true method => "MURMUR3" } # first filter grok { add_tag => ["valid", "elastic"] match => ["message", "%{DATESTAMP:log_date} \[%{DATA:service_version}\]\[%{DATA:tool_name}\] %{LOGLEVEL:level}\s+%{NOTSPACE:logger_name} %{GREEDYDATA:message}", "source", ".+" ] break_on_match => false overwrite => ["message"] } date { match => ["log_date", "yy-MM-dd HH:mm:ss.SSS"] } # ERROR grok { add_tag => ["ERROR"] match => ["level", "ERROR"] tag_on_failure => [] } # FATAL grok { add_tag => ["FATAL"] match => ["level", "FATAL"] tag_on_failure => [] mutate { remove_tag => ["valid", "beats_input_codec_plain_applied"] rename => { "[agent][name]" => "[site]" } rename => { "[agent][hostname]" => "[host]" } remove_field => ["[agent][ephemeral_id]", "[agent][id]", "[agent][type]", "[agent][version]", "input_type", "offset"] } } ``` } Hopefully this helps with understanding our setup. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 3, 2023, 9:50pm UTC](https://discuss.elastic.co/t/difference-in-sending-static-vs-live-feeding-logs-to-logstash-via-filebeat/346349/4 "2023-12-03T21:50:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.