Logstash forwarder, logstash, nxlog, redis, which one to use?


#1

when i started to play with ELK, i followed some article and setup elk like this: logstash -> redis -> logstash -> elastic search -> kibana. I'm prepared that logstash shipper will use more memory, but i wasn't prepared that it'll use a noticeable amount of CPU as well: the java.exe used 10-20% cpu (windows, monitors one folder with about 50 files inside updated constantly), which is a lot and out of consideration. I need something has a minimal footprint on the application server, which can be either windows or linux, and with the consideration of scalability and data integrity, what kind of architecture should I go for? thank you.


(Patrick Kik) #2

You are describing pretty much a standard setup that is widely used. You might want to look into Beats, one of the Elastic products.


#3

Looks like FileBeat is not yet ready.

About logstash forwarder, if logstash is too busy to take care of the incoming data , will forwarder slow down its pace in sending the data?


(Magnus Bäck) #4

Yes, logstash-forwarder will back off nicely.


#5

in this case, logstash-forwarder -> logstash -> elasticsearch vs. logstash->redis->logstash->elasticsearch, which one will you reommend? what are the pros/cons for each approach?


(Magnus Bäck) #6

I'd say it doesn't matter that much. Having a buffer in between means that messages can be shipped off of the leaf machine as quickly as possible, but you also get a more complicated setup. The main advantage of having a buffer is really if you have inputs that don't back off in a good way but that's not the case for you.


#7

I see, thank you.


(system) #8