Is logstash forwarder expected to be slower than logstash?


#1

At the beginning, I set up a logstash service on windows to monitor all files in one folder. The service runs fine, data feeding to elasticsearch through redis is up to date and good. The only drawback is the cpu and memory usage. Then i switched to use logstash-forwarder, yes, it uses little cpu, however, it's lagging behind greatly (data fed to elastic seach is not up to date), is this the expected behavior? LSF is slower than logstash?


(Mark Walkom) #2

It shouldn't be.
What does the stack architecture look like?


#3

For logstash:
logstash -> redis -> logstash -> elastic search
--Filter plugins are configured at shipper side

For LSF:
LSF -> logstash -> elastic search
--Filters are at logstash

maybe the box running logstash(indexer) and elastic search not powerful enough?


(Mark Walkom) #4

Are they going to the same ES instances?


#5

yes, they are.

Redis, indexer, and elastic search are on the same server. For the LSF configuration, I simply changed the indexer to use the lumberjack input rather than redis.


(system) #6