I was wondering if there is any way to obtain an InputStream when
running a get query using the Java API? The reason I am asking is
because I feel that loading documents with large attachments into
memory before forwarding attachments on to a 3rd party client is
somewhat inefficient. (i.e. downloading via HTTP)
What I would like to achieve ideally is obtain a stream to a document
attachment, pass that stream to the web server so that it forwards it
to the client, and base64 decode it on the fly. Intuition tells me
that this should be much more efficient than fetching the entire
document from ES, base64 decoding it, and then passing that in-memory
byte array to the web server as a stream.
Please correct me if I am missing something obvious. Thanks in advance!
Thats how it works currently..., no plans to change that as its quite a
hefty change (think about streaming across several nodes in the cluster),
and will require a different store for attachments.
I was wondering if there is any way to obtain an InputStream when
running a get query using the Java API? The reason I am asking is
because I feel that loading documents with large attachments into
memory before forwarding attachments on to a 3rd party client is
somewhat inefficient. (i.e. downloading via HTTP)
What I would like to achieve ideally is obtain a stream to a document
attachment, pass that stream to the web server so that it forwards it
to the client, and base64 decode it on the fly. Intuition tells me
that this should be much more efficient than fetching the entire
document from ES, base64 decoding it, and then passing that in-memory
byte array to the web server as a stream.
Please correct me if I am missing something obvious. Thanks in advance!
Thanks Shay! Just to clarify, when you say "that's how it works
currently", which part do you mean? Automatic base64 decoding as the
attachment is being streamed to the client?
Thats how it works currently..., no plans to change that as its quite a
hefty change (think about streaming across several nodes in the cluster),
and will require a different store for attachments.
I was wondering if there is any way to obtain an InputStream when
running a get query using the Java API? The reason I am asking is
because I feel that loading documents with large attachments into
memory before forwarding attachments on to a 3rd party client is
somewhat inefficient. (i.e. downloading via HTTP)
What I would like to achieve ideally is obtain a stream to a document
attachment, pass that stream to the web server so that it forwards it
to the client, and base64 decode it on the fly. Intuition tells me
that this should be much more efficient than fetching the entire
document from ES, base64 decoding it, and then passing that in-memory
byte array to the web server as a stream.
Please correct me if I am missing something obvious. Thanks in advance!
Yes, the full attachment is loaded to memory (_source), it does not get
another round of base 64 if you ask for _source, since its stored as is
(already in base64).
Thanks Shay! Just to clarify, when you say "that's how it works
currently", which part do you mean? Automatic base64 decoding as the
attachment is being streamed to the client?
Thats how it works currently..., no plans to change that as its quite a
hefty change (think about streaming across several nodes in the cluster),
and will require a different store for attachments.
I was wondering if there is any way to obtain an InputStream when
running a get query using the Java API? The reason I am asking is
because I feel that loading documents with large attachments into
memory before forwarding attachments on to a 3rd party client is
somewhat inefficient. (i.e. downloading via HTTP)
What I would like to achieve ideally is obtain a stream to a document
attachment, pass that stream to the web server so that it forwards it
to the client, and base64 decode it on the fly. Intuition tells me
that this should be much more efficient than fetching the entire
document from ES, base64 decoding it, and then passing that in-memory
byte array to the web server as a stream.
Please correct me if I am missing something obvious. Thanks in advance!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.