we use logstash because, we can filter the folders under log location and mutate the field and parse the logs accordingly.
my question is that, is it good idea to run logstash service on all the 60+ servers or can filebeat or any other apps can help me do achieve the above. Current architecture flow:
does filebeat has the feature of filtering the logs ? currently i observe that logstash consumes much resource on the servers also went through multiple blog which says the same that logstash consumes much resources.
Some architectures like yours can become either:
filebeat -> kafka -> logstash -> Elasticsearch -> kibana
filebeat -> logstash -> kafka -> logstash -> Elasticsearch -> kibana
etc. (Not going to list every possibility as they're not necessarly in the scope for your question.)
Important thing is, yes, if you can, it is often a good idea (and it is recommended) to run Filebeat on the app server instead of Logstash, for exactly the reason you mentioned which is performance. That's not because Logstash is bad and Filebeat is good, it's because they are different.
The question of whether you can do it in your usecase or whether the benefits would outweigh the cost is the "it depends" part that you'll need to look into, research and test.
Filebeat has a ton of features to fulfill the same usecases as logstash it terms of harvesting files on your app servers and doing some pre-processing and filtering. You should review the filebeat documentation, especially the section about the inputs definitions and the processors.
Then you'll see if you can use Filebeat to replace Logstash on the app servers in a beneficial way for your usecase. Most of the time, the answer is a resounding yes because even if filebeat cannot do everything you're currently doing with Logstash, it can most often do enough and you can move the tasks it cannot do to a Logstash further down in the pipeline and still end up with exactly what you had before.