My Setup: Have 50+ application servers which generates huge logs and filebeat ships them to multiple LS nodes and then gets shipped into 1 ES node[Actually 6 node ES cluster - 5 data nodes 1 client node]
In logstash conf, there are 2 output plugins, 1 is ES and other is file. So we feel every event writing in file takes a long time so filebeat queues up log files in application server. We temporarily stopped file plugin and the queue got drastically reduced in filebeat end.
We store only 1 month of app data in ES. For general auditing we need to store years of data in tapes. But since ES+file makes issues in our setup. Can we only push application log to ES and then extract and generate same kind of logs back from ES and store in a text file and then archive it. Is this possible? Kindly suggest if it is valid or suggest any better ideas?