Elasticsearch and large (binary) repository


(Gianni) #1

Hi,
I'm developing a document repository based on Elasticsearch 1.7
Each document is composed by list of metadata (fields) and a binary content.
The binary content is stored in a dedicated index in json format encoded in base64.
The dedicated indexes are assigned to dedicated nodes (with replica) in the Elasticsearch cluster (through rack id properties).
The growth in 2016 is expected to be around 10 TB.

I am aware of the possibility to store binary documents on a different data storage (Hadoop, Centera....),so my question is:
Can I continue on ES 1.7 or the new release 2.x (that natively integrated mapper-attachment plugin) provides greater security in terms of stability and maintenance?
What about upgrading from 1.7 to 2.x after the production release?

Regards,
Gianni


(Mark Walkom) #2

Updated releases always included improvements around performance and stability.

What about it specifically?


(system) #3