Is it possible to feed Logstash data into another Logstash instance that connects to Elastic?
In my configuration I will have many remote locations hooked up to expensive satellite connections that I want to send netflow and syslog data to a central ELK stack.
The problem is that data usage is too high. I'm considering running Logstash at every remote location to collect all logs and remove fields I don't need before sending it to the central Logstash instance that will do some further parcing such as performing DNS lookups on netflow data (can't do this on the remote host as it will use too much data).
Is this possible and will this reduce data usage? I don't know what format Logstash will send data in and whether it is possible to apply compression.
For log files I might be able to use Beats but I don't believe there is a Beat for netflow data.
Of course, if there is a better way to achieve what I want, please tell me
The pattern of... logstash -> message queue -> logstash -> elasticsearch... is fairly common.
It is really easy to use redis in this way, however redis provides no compression mechanism. You might want to look at Kafka instead. I haven't used this myself, but I notice the the Logstash Kafka output has the ability to compress the data before it is sent to Kafka. I am assuming that the Kafka input can then get the data uncompressed, but you will want to check that.
Which ever you choose, run the message queue at the central site and have the remote site send their data to the queue over the WAN/internet links. If data sent over the internet you will probably want to enable SSL for security reasons.
By lumberjack you mean in combination with filebeat or logstash-forwarder?
I would like to keep things as simple as possible, especially on the remote site as any maintenance there will be a pain in the rear. Redis and Kafka seem like overkill for what I want?
I did some reading but those all rely on log files to be saved somewhere? I'm working with netflow data so that would mean I either need to use another piece of software to save netflow to log files and then have lumberjack/forwarder/beats take care of it or run a full Logstash install and have that send it's data to the central Logstash server?
Does the latter make any sense?
Is there any documentation on how a setup like mine needs to be configured? I found plenty of information about how to configure logstash with lumberjack/forwarder but almost nothing on how to mutate data with lumberjack/forwarder/beats before sending it to logstash.
I want to delete and add some fields to my netflow and log files before sending it from the remote location into the logstash server for further processing (reverse DNS lookup on netflow data etc).
Logstash can receive and decode netflow data received over the network through the netflow codec plugin, after which you can process it using filters as usual and then send it on to another Logstash instance e.g. via the lumberjack protocol.
I did some reading but those all rely on log files to be saved somewhere? I'm working with netflow data so that would mean I either need to use another piece of software to save netflow to log files and then have lumberjack/forwarder/beats take care of it or run a full Logstash install and have that send it's data to the central Logstash server?
When we say lumberjack we're talking about the lumberjack input and output plugins to Logstash. Forget about logstash-forwarder (which originally was called lumberjack) and Beats.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
