With the td-agent are you referring to the fluentd agent? I'm not aware of the implementation they use so I would recommend to test if everything works as expected with Logstash itself first as this is what we test. I would recommend to update to alpha4 to make sure all the most recent fixes are included.
By default the scan_frequency is 10s in filebeat, so new records should be crawled latest after 10s. I would recommend you to have a look at the filebeat log file to see if there are some output issues in the logs which could explain the delays.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.