Vector Scoring

Well Ryan - You're a champ!
The binary approach is 7-8 times faster than my original approach.

I did it a little different:
in my ScriptEngineService I injected the BinaryDocValues reader (for my binary field) to the plugin, see setBinaryEmbeddingReader:

public SearchScript search(CompiledScript compiledScript, final SearchLookup lookup, @Nullable final Map<String, Object> vars) {
        final NativeScriptFactory scriptFactory = (NativeScriptFactory) compiledScript.compiled();
        final VectorScoreScript script = (VectorScoreScript) scriptFactory.newScript(vars);
        return new SearchScript() {
            @Override
            public LeafSearchScript getLeafSearchScript(LeafReaderContext context) throws IOException {
                script.setBinaryEmbeddingReader(context.reader().getBinaryDocValues(VectorScoreScript.field));
                return script;
            }
            @Override
            public boolean needsScores() {
                return scriptFactory.needsScores();
            }
        };
    }

and inside the run method in my plugin I got the byte array as follows:
final byte[] bytes = binaryEmbeddingReader.get(docId).bytes;
where docid is set at setDocument (my plugin implements LeafSearchScript)

Thanks for the help!