How does knn affects the score?

Elastic Stack Elasticsearch
1

Hello,

I saw that knn cannot be used in the _explain API in Kibana and wondered why.

I've seen that Reciprocal Ranking Fusion and Convex Combination both allow to compute a result from a classical query score and a knn match, but how can this be checked ? I'd love to see knn usable in the explain API to understand better.

Thank you,

2

Hello, would anyone know something about the matter ?

  • is it planned in the roadmap to add knn explainability in the explain api ?
  • is there some ressource or blog post I could find on the topic?
    Thank you,
    Yoann
3

@Yoann_Buzenet ,

knn is supported in explain already. You can just use the knn query if you are using the explain API. As for explain: true in the search request, it should tell you the individual vector score and if it was in the top-k or not for that query.

4

Hello Ben,
Thank you for your answer. I created a case to be reproduced to get the error on knn not supported in explain API.

Here it is:

  • Index creation:
PUT /knn_understanding
{
  "mappings": {
    "properties": {
      "embedding": {
        "type": "dense_vector",
        "dims": 384
      }
    }
  }
}

GET knn_understanding/_explain/fCYRjZIBeFiUHHY1tI2J
{
  "knn": {
    "field": "embedding",
    "query_vector": [
      -0.049409836530685425,
      0.004891033284366131,
      0.02810363657772541,
      0.05813274160027504,
      -0.02277800627052784,
      -0.08117515593767166,
      0.03672189638018608,
      -0.003742472967132926,
      0.09365282952785492,
      0.050613727420568466,
      0.013901573605835438,
      -0.020395124331116676,
      -0.007280553225427866,
      0.0022683285642415285,
      -0.0798516720533371,
      -0.020266443490982056,
      -0.027672559022903442,
      -0.0020338057074695826,
      -0.021304968744516373,
      -0.014463381841778755,
      0.013873877935111523,
      0.021126234903931618,
      -0.046334706246852875,
      0.019942348822951317,
      0.004430413246154785,
      0.07496818155050278,
      0.01780945248901844,
      0.027535229921340942,
      -0.16652490198612213,
      -0.05582409352064133,
      -0.024150878190994263,
      0.11059213429689407,
      -0.06544376164674759,
      -0.05248958244919777,
      -0.02786935865879059,
      0.012665773741900921,
      0.032295744866132736,
      0.006287253927439451,
      0.04858362302184105,
      0.004297235049307346,
      -0.037272483110427856,
      -0.13639354705810547,
      0.07986632734537125,
      0.05130545422434807,
      0.012659892439842224,
      0.004586723633110523,
      -0.016139421612024307,
      -0.03315325453877449,
      0.01820875145494938,
      0.06355420500040054,
      -0.07368788868188858,
      0.01303124614059925,
      -0.016112077981233597,
      0.01772332936525345,
      0.011353361420333385,
      -0.04840549826622009,
      0.005034640897065401,
      0.005646976176649332,
      0.0320860892534256,
      -0.11949650198221207,
      0.055256083607673645,
      -0.06408920139074326,
      -0.0685899630188942,
      0.05287231132388115,
      0.11058869957923889,
      0.013125206343829632,
      -0.029818184673786163,
      0.058537840843200684,
      -0.0017318298341706395,
      -0.05754252150654793,
      -0.0037360957358032465,
      0.0509151853621006,
      0.05079891160130501,
      0.06176655739545822,
      0.11007165163755417,
      -0.04189196601510048,
      0.05611620843410492,
      0.014012707397341728,
      0.11743518710136414,
      -0.059737835079431534,
      -0.007209470961242914,
      -0.05508410185575485,
      -0.10902736335992813,
      0.03140096738934517,
      -0.023651396855711937,
      0.02254081889986992,
      0.02056342363357544,
      0.0010580220259726048,
      -0.0675467923283577,
      0.03918212652206421,
      -0.08878873288631439,
      -0.06608942896127701,
      0.027788538485765457,
      0.0020840917713940144,
      -0.09848127514123917,
      0.04274114966392517,
      -0.05088504031300545,
      -0.08866234123706818,
      -0.011348066851496696,
      0.1275867074728012,
      -0.037042342126369476,
      0.0425901897251606,
      -0.05511794611811638,
      0.015774955973029137,
      0.09826533496379852,
      -0.09285881370306015,
      0.029072655364871025,
      -0.025634218007326126,
      0.030678875744342804,
      -0.000753039785195142,
      -0.005528176669031382,
      0.012077361345291138,
      -0.03817363828420639,
      0.012971200980246067,
      0.0861113965511322,
      0.07635430246591568,
      0.08410479873418808,
      0.041298262774944305,
      -0.0007305238978005946,
      -0.03311833366751671,
      -0.010606382042169571,
      0.004648401401937008,
      -0.11425226181745529,
      -0.02954685688018799,
      -0.0649784579873085,
      -0.03329478204250336,
      -0.0011585663305595517,
      -2.4508350081324278e-33,
      0.0316350944340229,
      -0.05173753574490547,
      0.020162351429462433,
      0.06597787886857986,
      0.051326408982276917,
      -0.032706670463085175,
      -0.024263229221105576,
      -0.023685527965426445,
      -0.0017541677225381136,
      0.0007936746114864945,
      -0.05776742100715637,
      0.06051736697554588,
      -0.06538413465023041,
      0.03647594526410103,
      0.08626258373260498,
      0.06327173113822937,
      -0.03350914269685745,
      0.017477653920650482,
      0.004297366365790367,
      -0.06454082578420639,
      -0.08032116293907166,
      0.021543487906455994,
      -0.0378723219037056,
      0.008934469893574715,
      0.02011488936841488,
      -0.01968550682067871,
      -0.035359419882297516,
      -0.05076120048761368,
      0.020596403628587723,
      0.029751403257250786,
      0.004540916066616774,
      0.11520950496196747,
      0.033320922404527664,
      -0.03450218588113785,
      0.057889603078365326,
      -0.062499918043613434,
      -0.03899291157722473,
      -0.02962207794189453,
      0.043336641043424606,
      0.06783860176801682,
      -0.04829634353518486,
      0.03092353790998459,
      -0.041549474000930786,
      0.0021326462738215923,
      0.018427260220050812,
      0.0223116185516119,
      0.09893038868904114,
      0.006302930414676666,
      -0.05646558851003647,
      0.088975690305233,
      -0.015586183406412601,
      -0.016343193128705025,
      0.02104244939982891,
      0.02683810330927372,
      0.01617836207151413,
      -0.01695675030350685,
      0.002295635873451829,
      -0.06167204678058624,
      -0.018849531188607216,
      0.004656618926674128,
      0.0322817824780941,
      0.13475686311721802,
      0.04503224045038223,
      -0.03957787901163101,
      0.04441124573349953,
      0.01955721527338028,
      0.04307923838496208,
      -0.021352173760533333,
      0.006152441259473562,
      -0.006302898749709129,
      -0.08052235096693039,
      0.08250440657138824,
      -0.01693098060786724,
      -0.0003445694746915251,
      0.005235031712800264,
      0.002937984187155962,
      0.08237919211387634,
      -0.012812864035367966,
      -0.09389758855104446,
      -0.006106135901063681,
      -0.024815933778882027,
      -0.03336304426193237,
      -0.02679702453315258,
      0.0192271638661623,
      0.02871078811585903,
      0.018955117091536522,
      -0.06065736338496208,
      -0.029904447495937347,
      -0.011691024526953697,
      -0.025439197197556496,
      -0.04619080573320389,
      -0.11162019520998001,
      0.0537695586681366,
      -0.05119529739022255,
      -0.09530068933963776,
      2.124256228851103e-33,
      -0.0758451297879219,
      -0.09364552050828934,
      -0.060507044196128845,
      0.046550050377845764,
      0.05473474785685539,
      -0.04278877377510071,
      -0.10247790068387985,
      -0.04446272924542427,
      -0.02499961294233799,
      0.0629647746682167,
      -0.10281739383935928,
      -0.006373060401529074,
      0.08628557622432709,
      0.010038319043815136,
      0.0498167984187603,
      -0.04578624665737152,
      -0.014049404300749302,
      -0.005578579381108284,
      -0.02636389434337616,
      0.04852703958749771,
      0.00042261360795237124,
      -0.008496541529893875,
      -0.015862051397562027,
      -0.040265291929244995,
      0.03237353637814522,
      0.06361255794763565,
      0.025288723409175873,
      0.019419299438595772,
      -0.08891844004392624,
      0.00151180534157902,
      0.02951311506330967,
      0.0008061358821578324,
      0.017357444390654564,
      0.007839015685021877,
      0.018047725781798363,
      0.1183367520570755,
      0.07228198647499084,
      -0.02813178487122059,
      0.027049094438552856,
      -0.07188857346773148,
      0.023648034781217575,
      0.028518086299300194,
      -0.08457920700311661,
      0.0813601016998291,
      -0.02336261235177517,
      -0.016507456079125404,
      0.06610347330570221,
      0.09352266788482666,
      -0.0036757360212504864,
      0.03184602037072182,
      -0.06957991421222687,
      0.017108725383877754,
      0.006004105322062969,
      -0.05464516207575798,
      0.030628317967057228,
      -0.029854673892259598,
      -0.09367018193006516,
      0.027028053998947144,
      -0.001451951451599598,
      0.05591083690524101,
      0.023848699405789375,
      0.021507469937205315,
      0.06981803476810455,
      0.06439214944839478,
      -0.07225064188241959,
      -0.05912204459309578,
      -0.03410935774445534,
      0.11967529356479645,
      -0.04684752970933914,
      0.003930406644940376,
      0.04005679115653038,
      0.02136135846376419,
      -0.04248519241809845,
      0.010312197729945183,
      -0.024079730734229088,
      0.010314052924513817,
      -0.022504057735204697,
      -0.03467293456196785,
      -0.04509783908724785,
      0.11609519273042679,
      -0.009770234115421772,
      -0.02358322963118553,
      0.023358827456831932,
      0.10350919514894485,
      -0.007483534049242735,
      0.09302595257759094,
      -0.002111472189426422,
      -0.03403764218091965,
      0.016529684886336327,
      -0.030004147440195084,
      0.016130944713950157,
      -0.03868066519498825,
      0.0104332584887743,
      0.05662151426076889,
      -0.04727974906563759,
      -1.357294454606972e-8,
      0.10642163455486298,
      -0.0013623692793771625,
      0.06026766076683998,
      -0.03951326385140419,
      0.024045240134000778,
      0.11666487157344818,
      0.0021126337815076113,
      0.022794438526034355,
      -0.04968615621328354,
      0.011026709340512753,
      -0.0664014220237732,
      0.02647843025624752,
      0.02762906812131405,
      0.06240236014127731,
      0.006782029755413532,
      -0.011187623254954815,
      0.03954063728451729,
      0.004930218681693077,
      -0.07038156688213348,
      0.06946661323308945,
      0.0376739539206028,
      0.044212184846401215,
      0.014575139619410038,
      -0.019005155190825462,
      -0.012388641014695168,
      -0.0029710594099014997,
      0.0694991946220398,
      -0.03681844845414162,
      0.01713174395263195,
      0.09619986265897751,
      -0.005801004823297262,
      0.024630054831504822,
      -0.11250384151935577,
      0.028666773810982704,
      0.006120818201452494,
      -0.042589426040649414,
      0.03790668398141861,
      -0.010988209396600723,
      -0.026690157130360603,
      -0.05288493633270264,
      0.013099487870931625,
      0.061343949288129807,
      -0.01793859899044037,
      0.05894609913229942,
      -0.0029380314517766237,
      -0.045418404042720795,
      0.07898731529712677,
      -0.03478022664785385,
      0.02704465575516224,
      -0.006895766593515873,
      0.016534635797142982,
      0.050051603466272354,
      0.006315434351563454,
      -0.010414527729153633,
      0.024025272578001022,
      0.061644546687603,
      -0.019397474825382233,
      0.08599943667650223,
      -0.08958672732114792,
      0.03130229935050011,
      0.054686710238456726,
      -0.03496289625763893,
      0.05289012938737869,
      -0.009723770432174206
    ],
    "k": 5,
    "num_candidates": 50,
    "boost": 0.5, 
    "filter": {
      "bool": {
        "filter": [
          {
            "range": {
              "tags.time.maxTime": {
                "lte": 30
            }
          }}
        ]
      }
    }
  }
}

And you get this error:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "request does not support [knn]",
        "line": 2,
        "col": 3
      }
    ],
    "type": "parsing_exception",
    "reason": "request does not support [knn]",
    "line": 2,
    "col": 3
  },
  "status": 400
}

Am I doing something wrong?

5

Yes. It's a top level knn request and not a knn query.

This should be like:

{
  "query": {
    "knn": {
      // ...
    }
  }
}