# Pull ES data and generate application logs **URL:** \ **Category:** Elasticsearch\ **Created:** [October 14, 2017, 5:05pm UTC](https://discuss.elastic.co/t/pull-es-data-and-generate-application-logs/104003 "2017-10-14T17:05:19Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![santhosh244](https://avatars.discourse-cdn.com/v4/letter/s/7ba0ec/32.png) [@santhosh244](https://discuss.elastic.co/u/santhosh244)\ **Post date:** [October 14, 2017, 5:05pm UTC](https://discuss.elastic.co/t/pull-es-data-and-generate-application-logs/104003/1 "2017-10-14T17:05:19Z")
Hi Team, My Setup: Have 50+ application servers which generates huge logs and filebeat ships them to multiple LS nodes and then gets shipped into 1 ES node[Actually 6 node ES cluster - 5 data nodes 1 client node] In logstash conf, there are 2 output plugins, 1 is ES and other is file. So we feel every event writing in file takes a long time so filebeat queues up log files in application server. We temporarily stopped file plugin and the queue got drastically reduced in filebeat end. We store only 1 month of app data in ES. For general auditing we need to store years of data in tapes. But since ES+file makes issues in our setup. Can we only push application log to ES and then extract and generate same kind of logs back from ES and store in a text file and then archive it. Is this possible? Kindly suggest if it is valid or suggest any better ideas? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 15, 2017, 4:57am UTC](https://discuss.elastic.co/t/pull-es-data-and-generate-application-logs/104003/2 "2017-10-15T04:57:19Z")
You can Logstash to do this, just setup a separate config pipeline to extract things. ---
**Author:** ![santhosh244](https://avatars.discourse-cdn.com/v4/letter/s/7ba0ec/32.png) [@santhosh244](https://discuss.elastic.co/u/santhosh244)\ **Post date:** [October 15, 2017, 5:56am UTC](https://discuss.elastic.co/t/pull-es-data-and-generate-application-logs/104003/3 "2017-10-15T05:56:12Z")
Hi Warkolm,/All, Sorry i forgot to inform the setup completely. I use filebeat -\> logstash -\> ES. In logstash output filter. First i have my event posting to ES and then to file plugin where i write back to a text file. So every event runs thought both the processes and then picks up next event. This operation is too costly. So i wanted a process such that i push my logs only to ES and store for 15-20 days and then generate a single file based on day wise index and store that txt file and archive it for years. Any suggestions on this please.. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 12, 2017, 5:56am UTC](https://discuss.elastic.co/t/pull-es-data-and-generate-application-logs/104003/4 "2017-11-12T05:56:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.