Get started with logstash

Elastic Stack Logstash
1

Ive been tasked with implimenting log stash as a means to get data into elastic and while Ive heard the name before I dont really know much more than that about it so to save me making some mistakes in the early stages wanted to get some advice on how best to go about it?

I hear some people impliment this on their elastic nodes while others do not are there pros abd cons to these approaches? I would have thought having log stash on its own box is better so it doesnt have an IO impact on elastic itself.

I have 3 nodes in elastic running Elastic 5.6.6 does this inform how many logstash nodes I should set up? Does logstash cluster in a simialr way to elastic?

Are there any othet dependencies I need to put in to get data from Logstash to Elasticsearch?

Are there any guides people can point me to to help me get set up quickly?

Are there any gotchas I shoulf be aware of going into this?

2

I hear some people impliment this on their elastic nodes while others do not are there pros abd cons to these approaches? I would have thought having log stash on its own box is better so it doesnt have an IO impact on elastic itself.

Segregating Logstash and Elasticsearch is typically a good idea but don't focus too much on deployment infrastructure at this point.

I have 3 nodes in elastic running Elastic 5.6.6 does this inform how many logstash nodes I should set up?

No.

Does logstash cluster in a simialr way to elastic?

No.

Are there any othet dependencies I need to put in to get data from Logstash to Elasticsearch?

Probably not.

3

Thanks for your feedback Magnus, do you have any guidance on the spec for the logstash nodes?

I'm looking to start with a VMs for logstash I'm guessing that's ok until I find that througput is such that they need dedicated disks or nics?

I have been reviewing some of the guidance on setting up logstash but I'm not seeing much by way of recommended specs for the servers? is there a minimum RAM reqirement? should I disable swap files to help performance?

advice much appreceted.

4

Thanks for your feedback Magnus, do you have any guidance on the spec for the logstash nodes?

It depends on the event rate and what you want Logstash to do with the events.

I'm looking to start with a VMs for logstash I'm guessing that's ok until I find that througput is such that they need dedicated disks or nics?

Logstash typically doesn't require much I/O and you'll saturate the CPUs long before the NICs.

is there a minimum RAM reqirement?

It does run in a JVM with a default max heap size of 1 or 2 GB so the very smallest machines will be too small.

5

The more I read the more applicatons I'm finding for it.

The primary focus at the moment is to log usage data for an API (where the API calls would output a log entry when called) and collate reports on that to better help us know where to direct our efforts in improving the API with information on what aspects are most of interest to end users so we make sure we market those popular aspects better.

Arugably a side project but I also data I move from RabbitMQ to HDFS & Hive on Hadoop which this might be able to help me with.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.