I have a large Document in elasticsearch(deployed in server) and i want get that document and store in local file without give more load on JVM- using streaming.
Is it doable or not, if yes how?
for example- in elasticsearch, Document size is 1GB and i have 500MB free memory available in JVM and i want few MB to be used out of 500MB in process.
Elasticsearch is not designed or optimised for documents that large so I do not think your scenario is possible/realistic.
Appreciated much, this is the current scenario i am in which so, Still searching
Do you really have documents if that size? If so, why?
just for example i have mentioned, but i will be having around 100 MB, but still i can not give load JVM, so i will allocate let say 1 to 10 MB for this process(may be some bucket size of buffer size as per solution)
There is no way to stream individual documents so having large documents like this requires extra heap. This is why it is not recommended. What makes these documents so large?
just json data only
Just text? No encoded binary data that inflates size?
yes, just text data
Could you share an example of a typical document you want to index? You can share on gist.github.com.
Thanks for your response, any sample Josn data you can assume/construct by repeating same data you can make 500000 lines
What kind of data is this? Can you break it up somehow? What do you do with the result once you find these documents in a search?
That is not going to help if I can't see your typical JSON.
I'd love to help and make suggestions but I can't without any idea of what you are doing.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.