We are planning to store in the index a large string(error log) with a classifier for future use in a machine learning project. The error log can get to the size of tens of megabytes and we are planning to store tens of thousands+ of documents. how should I define the index or even just the logText field such that it is compressed in elastic, or should I compress it before sending it to Elasticsearch and if so what should I set the field type to be.
Elasticsearch is not really built to handle very large documents efficiently, so if you are not indexing and searching the large text field I would recommend you store it elsewhere, e.g. S3, and just include a link in the document.
I would also recommend you upgrade to the latest version of Elasticsearch as the version you are using is very old and has been EOL for quite a while.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.