Have an all in one ELK setup, where Filebeat, Logstash and Elasticsearch reside on the same machine. Currently, I am able process live changes on designated log files, process them and display them through Kibana. To process offline log files, i.e., when someone just drops a tar file, I created another directory, in host and specified it in filebeat.yml, which is distinct from the ones that I use for live processing. This generally works fine. Problems I am having, wrt to offline processing are as follows:
Static offline log file is ingested at the same rate as the live ones, I suppose as dictated by bulk_max_size, and I would like it to be ingested at much faster rate
Looking under "Discover" tab, within Kibana, the X axist (@timestamp) is based on the current system, and I am wondering if I can change this to start from offline file's creation time. That is, if current time is 14:00; however, file's creation time is 09:00, I would like to see X axis starting from 09:00.
Here is my filebeat.yml:
# - /var/log/*.log
input_type: log document_type: syslog - paths: - /tmp_log/monitor/online/*.log - /tmp_log/monitor/offline/*.log document_type: monlog
rotateeverybytes: 10485760 # = 10MB
P.S. Kibana 4.4.2