# Binary fields do not support searching es 6.0.0-rc1

**URL:** <https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690>\
**Category:** Elasticsearch\
**Created:** [October 12, 2017, 9:55am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690 "2017-10-12T09:55:35Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Junon\_Lim](https://avatars.discourse-cdn.com/v4/letter/j/7993a0/32.png) [@Junon\_Lim](https://discuss.elastic.co/u/Junon_Lim)\
**Post date:** [October 12, 2017, 9:55am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/1 "2017-10-12T09:55:35Z")

</div>

I've followed the instructions given by rjernst at the below link but I'm getting "Binary fields do not support searching" error.

> [@Vector Scoring](https://discuss.elastic.co/t/vector-scoring/85227):
>
> Hi guys, I want to run K-Nearest-Nighbors on feature-vectors stored at ES. I wrote a plugin based on this plugin: https://github.com/MLnick/elasticsearch-vector-scoring I'm seeking to improve the query performance. right now it takes ~1 second - I wish to make it 10 times faster. Details: I have 2M documents in the index each document contains a 64 dimensions floats vector in a field named "embedding" this is it's mapping: "analysis": { "analyzer": { "payload\_analyzer": { "filter": "d…

Index Mapping:

```
{
  "testindex" : {
    "mappings" : {
      "testtype" : {
        "date_detection" : false,
        "properties" : {
          "body" : {
            "type" : "binary"
          }
        }
      }
    }
  }
}

```

The class looks like below. I've tried to put debugging points into the newinstance method but it's never hit.

```
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Map;
import java.util.function.Function;

//import org.apache.log4j.Logger;
import org.apache.lucene.index.BinaryDocValues;
import org.apache.lucene.index.LeafReaderContext;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.script.ExecutableScript;
import org.elasticsearch.script.SearchScript;
import org.elasticsearch.search.lookup.SearchLookup;
import org.elasticsearch.script.ScriptContext;
import org.elasticsearch.script.ScriptEngine;
import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.index.Term;
import org.apache.lucene.store.ByteArrayDataInput;
import org.apache.lucene.util.BytesRef;

public class MyExpertScriptEngine implements ScriptEngine {
	//final static Logger logger = Logger.getLogger(MyExpertScriptEngine.class);
	
    @Override
    public String getType() {
        return "expert_scripts";
    }

    @Override
    public <T> T compile(String scriptName, String scriptSource, ScriptContext<T> context, Map<String, String> params) {
        if (context.equals(SearchScript.CONTEXT) == false) {
            throw new IllegalArgumentException(getType() + " scripts cannot be used for context [" + context.name + "]");
        }
        // we use the script "source" as the script identifier
        if ("pure_df".equals(scriptSource)) {
        	//logger.debug("hello world");
        	System.out.println("hello world");
            SearchScript.Factory factory = (p, lookup) -> new SearchScript.LeafFactory() {
                final String field;
                final String term;
                boolean success;
                {
                    if (p.containsKey("field") == false) {
                        throw new IllegalArgumentException("Missing parameter [field]");
                    }
                    if (p.containsKey("term") == false) {
                        throw new IllegalArgumentException("Missing parameter [term]");
                    }
                    field = p.get("field").toString();
                    term = p.get("term").toString();
                }

                @Override
                public SearchScript newInstance(LeafReaderContext context) throws IOException {
                    //PostingsEnum postings = context.reader().postings(new Term(field, term));
                    BinaryDocValues binaryDocValues = context.reader().getBinaryDocValues(field);
                    
                    /*if (postings == null) {
                        // the field and/or term don't exist in this segment, so always return 0
                        return new SearchScript(p, lookup, context) {
                            @Override
                            public double runAsDouble() {
                                return 0.0d;
                            }
                        };
                    }*/
                    return new SearchScript(p, lookup, context) {
                        int currentDocid = -1;
                        @Override
                        public void setDocument(int docid) {
                            // advance has undefined behavior calling with a docid <= its current docid
                            /*if (postings.docID() < docid) {
                                try {
                                    postings.advance(docid);
                                } catch (IOException e) {
                                    throw new UncheckedIOException(e);
                                }
                            }*/
                        	try {
                        		success = binaryDocValues.advanceExact(docid);
							} catch (IOException e) {
								// TODO Auto-generated catch block
								e.printStackTrace();
							}
                            //currentDocid = docid;
                        }
                        @Override
                        public double runAsDouble() {
                            /*if (postings.docID() != currentDocid) {
                                // advance moved past the current doc, so this doc has no occurrences of the term
                                return 0.0d;
                            }
                            try {
                                return postings.freq();
                            } catch (IOException e) {
                                throw new UncheckedIOException(e);
                            }*/
                        	if(success) {
                        		try {
									BytesRef bytesRef = binaryDocValues.binaryValue();
									ByteArrayDataInput dataInput = new ByteArrayDataInput(bytesRef.bytes);
									int i =dataInput.readVInt();
									System.out.println(i);
								} catch (IOException e) {
									// TODO Auto-generated catch block
									e.printStackTrace();
								}
                        	}
                        	return 0.0d;
                        }
                    };
                }

                @Override
                public boolean needs_score() {
                    return false;
                }
            };
            return context.factoryClazz.cast(factory);
        }
        throw new IllegalArgumentException("Unknown script name " + scriptSource);
    }

    @Override
    public void close() {
        // optionally close resources
    }
}

```

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [October 12, 2017, 7:09pm UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/2 "2017-10-12T19:09:36Z")

</div>

Your code snippet formatting seems to be messed up, making it difficult to tell what is happening. You also appear to be missing the end of the compile method. Can you fix that so we can see the entire method/class?

---

<div class="post-metadata">

**Author:** ![Junon\_Lim](https://avatars.discourse-cdn.com/v4/letter/j/7993a0/32.png) [@Junon\_Lim](https://discuss.elastic.co/u/Junon_Lim)\
**Post date:** [October 13, 2017, 5:26am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/3 "2017-10-13T05:26:16Z")

</div>

Sorry for the bad format, corrected original post as above.

---

<div class="post-metadata">

**Author:** ![Junon\_Lim](https://avatars.discourse-cdn.com/v4/letter/j/7993a0/32.png) [@Junon\_Lim](https://discuss.elastic.co/u/Junon_Lim)\
**Post date:** [October 16, 2017, 8:04am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/4 "2017-10-16T08:04:12Z")

</div>

Another question is how does p and lookup get initialized in this line?  
SearchScript.Factory factory = (p, lookup) -\> new SearchScript.LeafFactory()

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [November 7, 2017, 7:38pm UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/5 "2017-11-07T19:38:23Z")

</div>

Sorry for the delayed response here, I missed the update. `(p, lookup) ->` is a lambda in java. SearchScript.Factory is an interface with a single method taking two parameters, which are `p` and `lookup`.

---

<div class="post-metadata">

**Author:** ![Junon\_Lim](https://avatars.discourse-cdn.com/v4/letter/j/7993a0/32.png) [@Junon\_Lim](https://discuss.elastic.co/u/Junon_Lim)\
**Post date:** [November 8, 2017, 2:29am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/6 "2017-11-08T02:29:12Z")

</div>

My main concern is on searching on binary fields, can you provide more insights on that (already formatted code above as you requested)?

Let me rephrase my question on (p,lookup). I know how lambda works, what I want to know is how they are initialized. You might want to point me to the source code somewhere.

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [November 10, 2017, 12:42am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/7 "2017-11-10T00:42:53Z")

</div>

Binary fields aren't really meant for searching. Their purpose is mainly to allow binary blobs to be returned in search results. I only say this to caution your expectations on gaining 10x performance.

Lambda's aren't "initialized", they are called. It is just a function that returns a new SearchScript.LeafFactory instance. That interface will be called once per lucene segment to create a SearchScript specific to that segment. If you are really curious about the inner workings, you probably want to look at ScriptScoreFunction.getLeafScoreFunction.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2017, 12:43am UTC](https://discuss.elastic.co/t/binary-fields-do-not-support-searching-es-6-0-0-rc1/103690/8 "2017-12-08T00:43:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.