# ELK Filebeat Offline and Online Processing **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 7, 2016, 10:36pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375 "2016-12-07T22:36:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Nastooh](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nastooh](https://discuss.elastic.co/u/Nastooh)\ **Post date:** [December 7, 2016, 10:36pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/1 "2016-12-07T22:36:25Z")
Hi Have an all in one ELK setup, where Filebeat, Logstash and Elasticsearch reside on the same machine. Currently, I am able process live changes on designated log files, process them and display them through Kibana. To process offline log files, i.e., when someone just drops a tar file, I created another directory, in host and specified it in filebeat.yml, which is distinct from the ones that I use for live processing. This generally works fine. Problems I am having, wrt to offline processing are as follows: 1. Static offline log file is ingested at the same rate as the live ones, I suppose as dictated by bulk\_max\_size, and I would like it to be ingested at much faster rate 2. Looking under "Discover" tab, within Kibana, the X axist (@timestamp) is based on the current system, and I am wondering if I can change this to start from offline file's creation time. That is, if current time is 14:00; however, file's creation time is 09:00, I would like to see X axis starting from 09:00. Here is my filebeat.yml: filebeat: prospectors: - paths: - /var/log/secure - /var/log/messages # - /var/log/\*.log Any thoughts? Cheers, P.S. Kibana 4.4.2 Logstash 2.2.4 ES: 2.4.1 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 8, 2016, 6:40am UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/2 "2016-12-08T06:40:25Z")
> Static offline log file is ingested at the same rate as the live ones, I suppose as dictated by bulk\_max\_size, and I would like it to be ingested at much faster rate What rate are you getting? Is it Filebeat, Logstash, or Elasticsearch that's the bottleneck? > Looking under "Discover" tab, within Kibana, the X axist (@timestamp) is based on the current system, and I am wondering if I can change this to start from offline file's creation time. That is, if current time is 14:00; however, file's creation time is 09:00, I would like to see X axis starting from 09:00. If the lines contain a timestamp you should use Logstash's date filter to parse it. ---
**Author:** ![Nastooh](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nastooh](https://discuss.elastic.co/u/Nastooh)\ **Post date:** [December 8, 2016, 10:46pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/3 "2016-12-08T22:46:50Z")
> [@magnusbaeck](#): > > Logstash's date filter Thank you for your reply. Regrading the ingestion rate, this is what I've done: 1. Ran online mode for around 1 minute, and observed that there were 5 items proceed and eventually displayed by Kibana, i.e., while under "Discover" tab saw 5 bars. 2. Stopped the online log generation, copied over the generated log file into /tmp\_log/monitor/offline/\*.log, and noted that again it takes ~ one minute to process and display the log data, which generated the same number of bars. I doubt if its a bottleneck issue, most likely, the issue is due to the fact that I am using the same configuration, for ES, Logstash and Filebeat, on both online and offline modes. I am wondering if it is possible to change relevant parameters, e.g., bulk\_max\_size, based on directory of origin, or other means, to allow for faster processing. Thank you for advice on date-filter. It would do nicely. Cheers, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 9, 2016, 6:42am UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/4 "2016-12-09T06:42:06Z")
> Ran online mode for around 1 minute, and observed that there were 5 items proceed and eventually displayed by Kibana, i.e., while under "Discover" tab saw 5 bars. Not sure what you mean here. The number of vertical bars in the Discover tab isn't the same as the number of processed messages. > I doubt if its a bottleneck issue, most likely, the issue is due to the fact that I am using the same configuration, for ES, Logstash and Filebeat, on both online and offline modes. I strongly doubt that offline or online "mode" has anything to do with this. ---
**Author:** ![Nastooh](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nastooh](https://discuss.elastic.co/u/Nastooh)\ **Post date:** [December 9, 2016, 4:29pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/5 "2016-12-09T16:29:45Z")
Thank you again. To be clear then are you saying that when a static log file is used as an input, it is in fact ingested and processed at once? (I.e., the observed behaviour, under Discover tab, is just a visual artefact.) Cheers, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 9, 2016, 4:38pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/6 "2016-12-09T16:38:22Z")
> To be clear then are you saying that when a static log file is used as an input, it is in fact ingested and processed at once? It can take seconds before a new file discovered, but apart from that it should get ingested quite quickly. ---
**Author:** ![Nastooh](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@Nastooh](https://discuss.elastic.co/u/Nastooh)\ **Post date:** [December 9, 2016, 4:39pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/7 "2016-12-09T16:39:55Z")
Got it. Thanks again Cheers, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 6, 2017, 4:40pm UTC](https://discuss.elastic.co/t/elk-filebeat-offline-and-online-processing/68375/8 "2017-01-06T16:40:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.