Hi. I have a situation where I need to associate supplementary data with documents indexed in Elasticsearch. This extra data won't be searched, it won't be retrieved except in select cases, and its size per document could range from a handful of bytes to tens of megabytes. It could be represented in a couple of fields consisting of a long list of numbers or a long string.
I am trying to determine whether adding this supplementary data to the corresponding index items as fields that are not indexed or included in _all is a viable approach. Are there any big costs to this, aside from the obvious (i.e. disk space, transfer time)? Elasticsearch is not being used as a primary data store in this scenario, and the extra data only has meaning in the context of the indexed items.
Using a separate data store for the associated data is certainly an option, however I would prefer to avoid the extra complexity that would introduce if at all practical.
Thanks in advance for any opinions or other feedback.
Jim