# Issue storing binary fields in 6.2.2

**URL:** <https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290>\
**Category:** Elasticsearch\
**Created:** [April 17, 2018, 6:08am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290 "2018-04-17T06:08:39Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\
**Post date:** [April 17, 2018, 6:08am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/1 "2018-04-17T06:08:40Z")

</div>

My application stores long text (\>32K) values in binary field. The code works well in 5.2.1. However when I upgraded to 6.2.2 I am seeing strange behavior. Blob of one document gets attached to other documents.

Here are steps to reproduce

1. Create index  
PUT /blob\_test  
{  
"settings": { "number\_of\_shards": 1, "number\_of\_replicas": 0 },  
"mappings": {  
"\_doc": {  
"dynamic": "false",  
"\_source": { "enabled": true },  
"properties": {  
"blob": { "type" : "binary", "store" : false, "doc\_values" : true }  
}  
}   
}  
}

2. Load data  
public static void main(String[] args) {  
try (RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200, "http")).build()) {  
StringBuilder reqBuf = new StringBuilder();  
for (int i = 0; i\< 10; i++) {  
String id = "id\_" + i;  
//add id to make blob distinct  
String blob = Base64.getEncoder().encodeToString((id + " and some text").getBytes());  
reqBuf.append("{"index" : {"\_id" : "" + id + ""}}\n");  
reqBuf.append("{"blob" : "" + blob + ""}\n");  
}  
HttpEntity entity = new NStringEntity(reqBuf.toString(), ContentType.APPLICATION\_JSON);  
restClient.performRequest("POST", "/blob\_test/doc/\_bulk",  
Collections.emptyMap(), entity);  
} catch (Exception e) {  
e.printStackTrace();  
}  
System.exit(0);  
}

3. Run Query  
POST /blob\_test/\_search?pretty  
{  
"\_source": true,  
"docvalue\_fields": ["blob"]  
}

Base64 encoded value in \_source and fields differ for many documents. Value in \_source is always correct but value in fields is wrong.

Same steps work in 5.2.1. It does not like \_doc as type. But changing to doc works.

Best regards  
Vinayak

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 17, 2018, 7:10am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/2 "2018-04-17T07:10:44Z")

</div>

Why using binary if it's a text that you want to search for?

---

<div class="post-metadata">

**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\
**Post date:** [April 17, 2018, 7:34am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/3 "2018-04-17T07:34:12Z")

</div>

Even though I have used small text in the example actual length is \> 32K. Keyword field with docvalue true throws exception DocValuesField is too large, must be \<= 32766.

Also, I don't search on this field. It's used only for display.

Vinayak

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 17, 2018, 7:54am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/4 "2018-04-17T07:54:29Z")

</div>

I see.

Why not using:

```
"blob": { "type" : "binary" }

```

And then use source filtering to just read the `blob` field from the `_source`?

---

<div class="post-metadata">

**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\
**Post date:** [April 17, 2018, 1:07pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/5 "2018-04-17T13:07:59Z")

</div>

Thanks David.  
That's a workaround I thought too. It will work for me in one index which is small but not for other which is significantly large (in TBs).

Large index data never gets updated. Both indices have lot of other fields that are indexed. I set \_source off in mappings and only rely on doc values. Even with best compression, we get good amount of saving by turning off "\_source" in the mappings. I kept \_source in this example for two reasons

1. demonstrate the issue that document reaches Elasticseach correctly but something fails during indexing or while reading it back.
2. Verify issue easily. It's not easy to spot difference in base64 encoded values.

Since my current approach works well in 5.2.1, I want make sure I have not missed something in mappings or in query specific to 6.2.2. Secondly using binary field with DocValues approach is not an anti-pattern or not being considered for deprecation.

Also, if it turns out to be 6.2.2 defect it would be nice to know if it can affect anything else.

Thanks  
Vinayak

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 17, 2018, 2:00pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/6 "2018-04-17T14:00:50Z")

</div>

TBH I'm not sure it's a very good thing to store binaries in elasticsearch.  
Anyway, you can probably do something like:

```
{
  "data": {
    "type": "binary",
    "store": true
  }
}

```

instead.

---

<div class="post-metadata">

**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\
**Post date:** [April 17, 2018, 4:00pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/7 "2018-04-17T16:00:40Z")

</div>

Thanks David.

> > TBH I'm not sure it's a very good thing to store binaries in elasticsearch.

It's not binary like a jpg. Raw value is actually string longer than 32K. We perform text analysis on this value. Since it's longer than 32K and I need to retrive complete string back, I wasn't able to store it as keyword.

I use two fields one of type text (used for searching) and one binary (for retrieving).

Coming back to your solution, store:true worked with a small test. I also tried storing text string larger than 32K. It worked too. I feel store:true is better option compared to enabling \_source. I will try with the real data.

I understand stored fields are slower to retrieve than docvalue fields ([What's the difference between "store" and "doc\_values" in field properties?](https://discuss.elastic.co/t/whats-the-difference-between-store-and-doc-values-in-field-properties/70045)). That should be okay.  
But does it have any implications on heap utilization compared to doc\_values? I don't search, sort, aggregate on binary field.

Thanks  
Vinayak

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 17, 2018, 4:18pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/8 "2018-04-17T16:18:25Z")

</div>

If you just store, I think that `store` is better than `doc_values`.  
`doc_values` is meant for sort and aggs.

But again, I'd store the binary content on a FS, distributed FS like HDFS, CouchDB, or whatever... And just put the link to the datastore in elasticsearch.

You'll pay a price in IO at some point when merging of segments will happen.

My 2 cents

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 17, 2018, 4:45pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/9 "2018-04-17T16:45:55Z")

</div>

There is a bug here. I opened [https://github.com/elastic/elasticsearch/issues/29565](https://github.com/elastic/elasticsearch/issues/29565).

I agree that `store` would be a better fit than doc values for your use-case. Doc values might look faster but they might not scale as well when you have lots of data and want to request multiple fields per document. This is due to the fact that stored fields have all values for a given document in a single place while doc values have conceptually one different file per field.

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 19, 2018, 9:38am UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/10 "2018-04-19T09:38:48Z")

</div>

A bugfix has been pushed and will be available in 6.3: [https://github.com/elastic/elasticsearch/pull/29567](https://github.com/elastic/elasticsearch/pull/29567).

---

<div class="post-metadata">

**Author:** ![Vinayak\_Sapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinayak_sapre/32/45939_2.png) [@Vinayak\_Sapre](https://discuss.elastic.co/u/Vinayak_Sapre)\
**Post date:** [April 22, 2018, 12:54pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/11 "2018-04-22T12:54:12Z")

</div>

David and Adrien,

I tested with a larger dataset. Using stored value instead of docvalue solved the problem.

Thanks for the solution.

Vinayak

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 20, 2018, 12:54pm UTC](https://discuss.elastic.co/t/issue-storing-binary-fields-in-6-2-2/128290/12 "2018-05-20T12:54:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.