Hello Elastic Community,
I’m working on an application that fetches logs from Elasticsearch and writes them to an XML file. However, I’m running into an issue when managing large volumes of data, and I’m seeking advice on the best way to handle it. Here's my sample scenario:
- I am fetching 2000 logs from Elasticsearch.
- I am writing these logs to XML files with a file size limit of 100 MB.
- If the size of an XML file exceeds 100 MB, I create a new file (e.g.,
logs_001.xml,logs_002.xml,logs_003.xml). - Let's say while writing the third file, I stop after writing the 1800th log.
- When I restart my application, I want to continue fetching the remaining 200 logs that were not written yet (from log 1801 to 2000) and continue writing them to the XML.
How can I efficiently implement this in .NET? Specifically, what’s the best way to tell Elasticsearch to start fetching logs from where I left off in my previous session?
Thank you in advance for your help!