Evade base64 binary data encoding

Elastic Stack Elasticsearch
1

Hi!

Is there any way to store binary data fields without base64 encoding? I have a bunch of such fields in each document and it will be costly for me to encode/decode them every time. I don't want to index them. Or is there a possibility for Elasticsearch plugin to get low-level Lucene API interface? Because, as far as I know, they have StoredField:
https://lucene.apache.org/core/5_3_1/core/org/apache/lucene/document/StoredField.html#StoredField(java.lang.String,%20byte[])

2

Hey,

as Elasticsearch also supports CBOR, which would allow you to sent binary data to elasticsearch as byte streams. That way you could circumvent the conversion on the cost that your client has to speak this protocol.

--Alex

3

Hi, Alexander!

Thanks for your answer. Can you please point me out to some examples with Elastic and CBOR?

4

Hey,

I think the only way to look at are the tests in the elasticsearch repository. The following tests are mentioning CBOR

  • CborFilteringGeneratorTests
  • CborXContentTests
  • JsonVsCborTests
  • RestTableTests
  • XContentFactoryTests
  • XContentTypeTests

also checking out the underlying implementation (jackson) that serializes/unserializes CBOR makes sense.

Hope this helps.

--Alex

5

Thanks, will check it out!