Straightforward hardware question

Elastic Stack Kibana
1

Hi all,

I apologise if this question is naive, but finding a definitive answer has absorbed too much time alread so I'm putting it to the community :smile:

I am building a SIEM system based around the ELK stack. The system accepts enormous volumes of data, 30-50Gb per day for starters, and is expected to be able to index and search as much of that data as possible and as fast as possible.

The current design calls for boxes with large amounts of RAM, in order to avoid 'courier fetch failure' errors when the indexes under search grow too large but, I have a very simple question:

As the index size increases to very large sizes (Terabytes), which component requires more RAM to return searches? ElasticSearch or Kibana?

When a courier fetch failed error is seen, which component has run out of memory? Is it the ElasticSearch nodes beneath, or is it the Kibana search engine itself?

The current design calls for almost a Terabyte of RAM split across three machines; one Kibana, two ElasticSearch. Each ElasticSearch box will run multiple instances, each with 32Gb of RAM each.

Will I be able to search a larger index by applying more RAM to the Kibana search box (I can easily give it 768Gb) or by adding more and more ElasticSearch nodes?

Again, apologies if this is well known around here :blush:

2

KB just talks to ES, so ES is where you want all your resources.

3

So a Kibana install can return results from an elasticsearch cluster of enormous size without having hundreds of GB of RAM itself?

I understand it's needed to cache results in RAM, but beyond that it can search on any size of data set?

4

KB just takes the query response and displays it. I don't know how much memory it'd take for massive data sets though, maybe one of the devs can comment further on that.