Logstash Capacity planning

Elastic Stack Logstash
1

I have an requirement to pull and process 1 TB of data per data using logstash and push it to elastic search.

  • how to calculate the network bandwidth for input data to logstash.

  • how to calculate the network bandwidth for output data from logstash.

  • how to calculate cpu/RAM/storage for logstash.

Any help is appreciated.

-Saravanan

2

how to calculate the network bandwidth for input data to logstash.

You'll have some protocol overhead but not much. Ignoring that the data rate is obviously going to be the amount of data divided by the processing time.

how to calculate the network bandwidth for output data from logstash.

That depends on how you're going to process the data, i.e. what will 1 TB input data become when Logstash is done with it?

how to calculate cpu/RAM/storage for logstash.

CPU requirements are highly dependent on what your events look like and what filters you use to process them (so you need to measure). RAM-wise you should be fine with a couple of GB, and storage needs are pretty much zero except Logstash's own logs (and the log retention policy you choose).

3

Thanks magnus :slight_smile:

If I convert raw 1TB data to json without any filtering and push same 1 TB data to elasticsearch ?

4

If I convert raw 1TB data to json without any filtering and push same 1 TB data to elasticsearch ?

As I said you should measure yourself with a reasonably small sample size of actual data.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.