Use logstash as a central logging server for log files exported from various devices

Elastic Stack Logstash
1

Hi All,
I want to use logstash as a central logging server for storing log files exported from various devices. The log files can contain structured as well as un-structured data. Also I would like to store binary files as well which can be gzipped before sending to logstash. Can someone suggest what should be the pipeline configuration to achieve this?

Thanks,
Arinjay

2

Hello,

Logstash does not store anything, it is an ETL tool that process data, it has many inputs, filters and outputs that enables you to receive/read logs from different sources, transform them and send to different outputs where they will be stored.

Probably you want to store your logs in Elasticsearch to be able to query your data.

This is no problem for logstash, but you may need to parse your logs to make it easier to search your data.

It does not make much sense to store binary data in elasticsearch as it is an search engine, it will just wast disk space.

Your question is too broad, you need to provide examples of what devices you want to monitor, what your data looks like, there are plenty of posts in the forum as well with many examples.

3

Thanks @leandrojmp for the response. The requirement to store binary data is to be able to fetch process core files when a process crashes on the device. This obviously is not searchable in Elastic but is needed to analyse the reason behind the crash. The associated process logs stored in Elastic will help in this.

Another example of binary data is to periodically pull CPU, memory and disk I/O metrics from the device which an application like Linux perf can then use it to generate Flame graphs.

I have come across metric beats module which can ship this type of device metrics to logstash, but am not sure how to configure logstash to handle it.

Thanks,
Arinjay

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.