when i started to play with ELK, i followed some article and setup elk like this: logstash -> redis -> logstash -> elastic search -> kibana. I'm prepared that logstash shipper will use more memory, but i wasn't prepared that it'll use a noticeable amount of CPU as well: the java.exe used 10-20% cpu (windows, monitors one folder with about 50 files inside updated constantly), which is a lot and out of consideration. I need something has a minimal footprint on the application server, which can be either windows or linux, and with the consideration of scalability and data integrity, what kind of architecture should I go for? thank you.
You are describing pretty much a standard setup that is widely used. You might want to look into Beats, one of the Elastic products.
Looks like FileBeat is not yet ready.
About logstash forwarder, if logstash is too busy to take care of the incoming data , will forwarder slow down its pace in sending the data?
Yes, logstash-forwarder will back off nicely.
in this case, logstash-forwarder -> logstash -> elasticsearch vs. logstash->redis->logstash->elasticsearch, which one will you reommend? what are the pros/cons for each approach?
I'd say it doesn't matter that much. Having a buffer in between means that messages can be shipped off of the leaf machine as quickly as possible, but you also get a more complicated setup. The main advantage of having a buffer is really if you have inputs that don't back off in a good way but that's not the case for you.
I see, thank you.