I need a setup to monitor logs from 100 different machines. I checked beats framework and it seems i cannot install a package in the machine. Also does beats support Multi-line? Also as per this page https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html , its better to send the data from logs to a message layer, then process it and then send to elastic search. I couldn't find a kafka output for beats.
So If i run very minimal logstash instance on tte 100 machines, with no processing and indexing, (Just throw the data to kafka with a multi-line filter added), will it have significant performance impact compared to using beats?
Filebeat does not (currently) do multiline - each line is sent as an individual event.
If you want to send beats to Kafka, you will have to use Logstash as an interim.
Your best solution if possible would be to ensure that your application writing the logs creates single-line (json encoded) log lines so that you can use filebeat to transport to logstash, and from there on to Elasticsearch , kafka etc.
First of all thanks for the prompt reply.
The application cannot be changed to log into a singe line.
If i run logstash instances itself in the 100 machines (which does basic forwading) will it impact the performance of the machine compared to using beats?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
