I want to save about 50TB/day's data from logstash and search them with elasticsearch. Is there any good solution for it?
According to offical introductions, I only can think out the below three ways, but I cannot make sure wethere there are correct, because of the large data. Could anyone help me please? Thanks a lot.
1)Logstash outputs data into elasticsearch.
2)Elasticsearch uses snapshot to keep a backup in hadoop(hdfs), and can restore it.
1)Logstash outputs data into hadoop(hdfs).
2)Mount hdfs as a local fs with NFS.
3)Elasticsearch the local hdfs.
1)Logstash output data into hadoop(hdfs).
2)Create a table and and external table which is STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler' in Hive.
3)Load data into table.
4)Elasticsearch can index the data.