Need help on logging solution

Elastic Stack Logstash
1

At present, we have ELK stack running on Ubuntu and filebeat on windows to collect logs from different applications running on it. If we want to send logs directly to logstash instead of using filebeat as data shipper, Do we need to install logstash on each windows machine to capture the logs or is it possible to write directly to the centralised logstash pipeline ?

I read about logstash forwarder as a data shipper and its been replaced by Beats(light weight shipper). But I need help on choosing the right solution to our environment. With filebeat, we need to add manually some custom fields which we think can be exported by the application/service itself. For example - service name, service version, app id etc.

2

You need to install or maintain some kind of log shipper on the Windows machines, be it Logstash or Filebeat.

With filebeat, we need to add manually some custom fields which we think can be exported by the application/service itself. For example - service name, service version, app id etc.

Perhaps that information could be put into the Filebeat configuration during deployment of a new application version? Otherwise the information needs to be made available in the logs themselves.

3

Thanks for the quick reply @magnusbaeck. Yes, we already put those information in filebeat configuration. And this will be changed everytime new version is deployed. Other possibilibity is to add in the logs themselves as you suggested.

If I am using Logstash instead, How can I achieve the same? My team here is concerned about updating the configuration each time. Is there any other alternatives to automatically update the these fields?

4

If you use Logstash you could have a translate filter that looks things up in an external file (e.g. YAML or CSV) that's automatically reloaded.

5

Thats a good option! Perhaps that information could be put into the Filebeat configuration during deployment of a new application version? Otherwise the information needs to be made available in the logs themselves
So there is no other alternatives other than the above to auto-update config if we use filebeat right ? Just to confirm.

6

I can't think of anything else right now.

7

Thank you @magnusbaeck.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.