At the beginning, I set up a logstash service on windows to monitor all files in one folder. The service runs fine, data feeding to elasticsearch through redis is up to date and good. The only drawback is the cpu and memory usage. Then i switched to use logstash-forwarder, yes, it uses little cpu, however, it's lagging behind greatly (data fed to elastic seach is not up to date), is this the expected behavior? LSF is slower than logstash?
It shouldn't be.
What does the stack architecture look like?
For logstash:
logstash -> redis -> logstash -> elastic search
--Filter plugins are configured at shipper side
For LSF:
LSF -> logstash -> elastic search
--Filters are at logstash
maybe the box running logstash(indexer) and elastic search not powerful enough?
Are they going to the same ES instances?
yes, they are.
Redis, indexer, and elastic search are on the same server. For the LSF configuration, I simply changed the indexer to use the lumberjack input rather than redis.