Function_score is so slow . how can i slove it?

Elastic Stack Elasticsearch
1

just like this . : Simple Function Score Query Shows Very Poor Performance

I use the version es 6.8. I write a rescore plugin in java and the speed is so slow .
I rescore need to fetch one field to calculate the score like bm25.

I find it is slow because of the code like :
String sentence=(String)leafLookup.source().get("fieldname");

when I replace the code as a constant string is it faster.how can i solve it ?

2

The key code here,very simple:

package com.esplugin.demo;

import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.index.Term;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.plugins.ScriptPlugin;
import org.elasticsearch.script.ScoreScript;
import org.elasticsearch.script.ScoreScript.LeafFactory;
import org.elasticsearch.script.ScriptContext;
import org.elasticsearch.script.ScriptEngine;
import org.elasticsearch.search.lookup.SearchLookup;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Collection;
import java.util.Map;
import java.util.Set;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.Scorer;

import com.pingan.llx.es.util.MyBM25FSimilarity;

public class MyScriptEngine implements ScriptEngine {
private final static Logger logger = LogManager.getLogger(MyRankPlugin.class);
public static String type="MyRankPlugin";
@Override
public String getType() {
return type;
}

@Override
public <T> T compile(String scriptName, String scriptSource,
                     ScriptContext<T> context, Map<String, String> params) {
    if (context.equals(ScoreScript.CONTEXT) == false) {
        throw new IllegalArgumentException(getType()
                + " scripts cannot be used for context ["
                + context.name + "]");
    }
    // we use the script "source" as the script identifier
    if (type.equals(scriptSource)) {
        ScoreScript.Factory factory = PureDfLeafFactory::new;
        return context.factoryClazz.cast(factory);
    }
    throw new IllegalArgumentException("Unknown script name "
            + scriptSource);
}

@Override
public void close() {
    // optionally close resources
}

private static class PureDfLeafFactory implements LeafFactory {
    private final Map<String, Object> params;
    private final SearchLookup lookup;
    private  MyBM25FSimilarity myBM25=new MyBM25FSimilarity();

    private PureDfLeafFactory(
            Map<String, Object> params, SearchLookup lookup) {
        if (params.containsKey("query") == false) {
            throw new IllegalArgumentException(
                    "Missing parameter [query]");
        }
        if (params.containsKey("f") == false) {
            throw new IllegalArgumentException(
                    "Missing parameter [f]");
        }
        this.params = params;
        this.lookup = lookup;
    }

    @Override
    public boolean needs_score() {
        return false;  // Return true if the script needs the score
    }

    @Override
    public ScoreScript newInstance(LeafReaderContext context)
            throws IOException {
            /*
             * the field and/or term don't exist in this segment,
             * so always return 0
             */
            return new ScoreScript(params, lookup, context) {

                @Override
                public double execute() {
                    try {

                        String query = params.get("query").toString();
                        String f=params.get("f").toString();
                        //logger.info("this is query value2 : {}  {}", query,f);

                        String sentence = (String) lookup.source().get(f);
                        //logger.info("this is sentence : {}", sentence);

                        double score = myBM25.score(query, sentence);
                        //logger.info("this is query value2 : {},{},{},{}", query,sentence,String.valueOf(score),String.valueOf(rawScore));
                        //this.setScorer(score);
                        return score;
                    }catch(Exception e){
                        e.printStackTrace();
                    }
                    return 0;
                }
            };
        }
    }

}

3

about 7-10 times slower than without a Function Score Script

4

Using data from source means the document need to be loaded, which will take longer.
Lower performance us therefore IMHO expected.

How long is the field?

5

the field is short text no more than 50 chars.

IMHO? how can i use the api ?

6

Have you got the field indexed as keyword so it would be available as a doc value?

7

No the field I got is indexed not as keyword but seg with default segment.
I tried String a=((ScriptDocValues.Strings)this.getDoc().get(f)).getValue() ; to get the field data but not sucess.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.