# How to read indexed binary field data from doc values (in custom Query plugin)

**URL:** <https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387>\
**Category:** Elasticsearch\
**Created:** [July 27, 2023, 6:19am UTC](https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387 "2023-07-27T06:19:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\
**Post date:** [July 27, 2023, 6:19am UTC](https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387/1 "2023-07-27T06:19:40Z")

</div>

Hi!

We're trying to write a custom query-plugin for Elasticsearch where we would use binary data for calculating scores (disclaimer: I've never written one before). Each document can have multiple vectors, so we cannot use the new dense vector type.

## Binary mapping

We've made a binary mapping for the vector data:

```json
"viewVectors": {
  "type": "binary",
  "doc_values": true,
  "store": true
}

```

## Indexing binary data as base64 encoded string

And then we index the binary data in base64 encoded format:

```scala
val binaryData: Array[Byte] = calculateVectors(???)
document.viewVectors = java.util.Base64.getEncoder().encodeToString(binaryData)
// And index this via REST APIs

```

## Trying to read binary data from DocValues

However, the problem is that when we're trying to get the binary data from `context.reader.getBinaryDocValues(fieldName).binaryValue()` the binary data is completely different than what we put there (we use Scala, but I hope it's pretty clear to Java developers as well):

```java
case class VectorDistanceQuery(fieldName: String, searchVectors: List[Array[Long]]) extends Query {
  override def toString(field: String): String = s"VectorDistanceQuery(fieldName=$fieldName)"
​
  override def createWeight(searcher: IndexSearcher, scoreMode: ScoreMode, boost: Float): Weight = {
    new ConstantScoreWeight(this, boost) {
      override def scorer(context: LeafReaderContext): Scorer = {
        val values = context.reader.getBinaryDocValues(fieldName)
        if (values == null) {
          null
        } else {
          val iterator = new TwoPhaseIterator(values) {
            override def matches: Boolean = {
              // PROBLEM: The problem is when we read the bytes the data is not what we have indexed? 🤨
              val binaryData = values.binaryValue().bytes // Shouldn't this be the same as what we put in foobar.viewVectors (without base64 encoding)
              val documentSearchVectors = decodeFromBinary(bytes)
              ???
            }
​
            override def matchCost = 42 // TODO: what should it be?
          }
          new ConstantScoreScorer(this, boost, scoreMode, iterator)
        }
      }
​
      override def isCacheable(ctx: LeafReaderContext): Boolean = DocValues.isCacheable(ctx, fieldName)
    }
  }
}

```

What are we missing? Why is the binary data read from `context.reader.getBinaryDocValues(fieldName)` different from the indexed base64-encoded data `java.util.Base64.getEncoder().encodeToString(binaryData)`?

— Help appreciated! 🙇‍♂️

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 24, 2023, 6:19am UTC](https://discuss.elastic.co/t/how-to-read-indexed-binary-field-data-from-doc-values-in-custom-query-plugin/339387/2 "2023-08-24T06:19:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.