Storing Very Large Text Field in Elasticsearch

[stephenb (Stephen Brown)]

Hi @kdwolf

I'm a little confused. Are you referring to a single token that is 40-50MB, or a text field containing multiple strings/tokens (this seems to be what you are implying)? If so, that is fine to store 40-50MB in a single text field in Elastic.

Perhaps you are confusing the single token limit in Lucene, which is 32K

Lucene still has a document limit (field limit) of about 2GB.

That said, there are other areas of concern, like actually returning the data due to HTTP limits, etc.

Perhaps take a look at this thread

Maximum characters limit text fields in elasticsearch

I think this documentation largely answers your question: General recommendations | Elasticsearch Guide [8.17] | Elastic Other fields (like keyword) have further limitations, but for text fields I think it's however much data you can send to ES up to the http limit which is about 100MB or if configured higher we eventually disallow blobs or text fields above 2GB. In practice if you consider for instance Moby Dick I think that's about ~2MB of text. So it's probably safe that anything human-con…

Minor Side Note: Most of the text in a .mobi file is binary so not sure if that is just and example ... The Header etc is text but the actual text of the document is binary AFAIK.