Elasticsearch (ES), Filebeat, Logstash, and Kibana for backup and restore purposes

Elastic Stack Elasticsearch
1

Could you clarify the following items for me? Could you please help me with this?

  • We have been using Elasticsearch (ES), Filebeat, Logstash, and Kibana for backup and restore purposes.
  • All components will run exclusively on-premises. Since our client environment restricts the use of cloud services, everything must run on-premises.
  • The Elasticsearch (ES) setup is in non-cluster mode, storing data on a single Windows server.
  • The data is expected to grow from 1 GB to more than 1 TB. How should disk storage on a single server be managed to accommodate this growth?
  • What is the best approach to implement data storage transitioning from hot to frozen mode? The frozen data storage might be on another Windows server. How can hot and frozen data storage be effectively implemented in an on-premises environment?
  • How can data be retrieved from the frozen data storage in case it is not available in the hot storage?
  • Can searchable snapshots be implemented without Kibana?

We are seeking technical expert advice to implement the ELK stack in our environment. Could you please help me with this.

Regards,
Michael Mathan S

2

For the frozen tier, you are going to need a commercial license.
So I'd enter in discussion with the sales team and a solution architect will help you with this.

The Elasticsearch (ES) setup is in non-cluster mode, storing data on a single Windows server.

I'm not sure we are "supporting" if the cluster is not viable. I mean that a minimum of 3 nodes is expected for a cluster.

What is the best approach to implement data storage transitioning from hot to frozen mode?

Using ILM.

The frozen data storage might be on another Windows server. How can hot and frozen data storage be effectively implemented in an on-premises environment?

You would need to mount the network drive as a shared file system.

How can data be retrieved from the frozen data storage in case it is not available in the hot storage?

This is how the searchable snapshot feature is working out of the box with Partially mounted index. See Searchable snapshots | Elasticsearch Guide [8.13] | Elastic

Can searchable snapshots be implemented without Kibana?

Yes. Everything is API driven.