Dense_vector slow in ES 9.1.3

We’ve been using ES 8.9.0 and are looking to upgrade, so I started testing performance with 9.1.3 and found it to be very slow. So, I went back to 8.19.3 and found that it’s performance is fine. But we’d like to not be stuck on ES 8, so I’m wondering if there’s anything we can do to fix the performance.

Our use case involves getting the cosine sim for multiple vectors and feeding the those values into a non-linear model to generate a final score. Because of this, we can’t use kNN so we set the “index”: false for the dense_vectors.

To simplify the test, I created an index with just two vectors fields:

{
"mappings": {
"properties": {
"vectorOne": {
"type": "dense_vector",
"dims": 768,
"index": false
},
"vectorTwo": {
"type": "dense_vector",
"dims": 768,
"index": false
}
}
},
"settings": {
"index": {
"mode": "standard",
"number_of_shards": "4",
"number_of_replicas": "0"
}
}
}

Then I loaded 250k documents with the two vectors and used the following query for performance testing:

{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "Math.max(0, cosineSimilarity(params.vectorOne, 'vectorOne') + cosineSimilarity(params.vectorTwo, 'vectorTwo'))",
"params": {
"vectorOne": [...],
"vectorTwo": [...]
}
}
}
}
}

In 8.19.3, this query averages 248 ms. In 9.1.3, it averages 683 ms. This is running on a cluster with a single AWS r7a.xlarge data node, 4 GB JVM heap (~28 GB available for disk cache). Are there any settings that could improve the performance? I tried “mode”: “standard” to see if switching away from LogDB would help, but it didn’t seem to make a difference. I tried adding more shards, but that made performance worse. I tried adding -Dvector.rescoring.directio=false in case it was using DirectIO instead of going through the disk cache, but that had no effect.

Hi @glund0,

Welcome! I don't see any GitHub issues from 8 to 9 (only this one from 8.6 to 8.18). Can you share the JVM settings and the output of the profiler for the above query?

Let us know.

Here’s the 9.1.3 profile info (I removed the query details after the first shard since the query is large due to the vectors):

{
        "shards": [
            {
                "id": "[C6ztsNfJS56-Nbzhk-Co1Q][vectest][0]",
                "node_id": "C6ztsNfJS56-Nbzhk-Co1Q",
                "shard_id": 0,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "script_score (*:*, script: {Script{type=inline, lang='painless', idOrCode='Math.max(cosineSimilarity(params.vectorOne, 'vectorOne') + cosineSimilarity(params.vectorTwo, 'vectorTwo'), 0)', options={}, params={vectorOne=[0.013173203, 0.0052297856, 0.21519354, -0.093842305, -0.01095355, -0.29436445, 0.06493544, 0.05938409, -0.17600165, -0.20461452, -0.015728287, 0.0046440605, -0.030841334, 0.122089006, -0.3034737, 0.037594393, -0.10002559, 0.11257306, -0.111819915, -0.12071488, 0.020595897, 0.15776865, 0.052580394, -0.09745094, 0.079588, -0.16879044, -0.034594744, -0.050949633, 0.08187252, -0.16822045, -0.052350275, 0.048628043, 0.07935129, 0.03799321, 0.10957204, 0.028511189, 0.17220527, -0.12096245, -0.066655464, 0.08931657, -0.054182395, 0.12907726, -0.03926433, 0.058102854, -0.17350669, 0.03771998, -0.08872518, 0.011460013, -0.01628334, 0.07906166, -0.05854619, -0.05143622, -0.09604408, -0.09826985, 0.045209, 0.05044051, 0.02117565, -0.057305027, 0.04241245, 0.043347206, 0.07159769, 0.018550502, 0.04424336, 0.06912926, -0.009729412, 0.03211654, 0.04782096, -0.08818417, -0.035109524, 0.018651119, 0.03159385, 0.022730356, -0.020502992, -0.066785574, -0.029437877, 0.018001571, -0.08194266, -0.056691643, 0.027324012, 0.025953125, 0.058490414, 0.031390805, -0.04929738, 0.02078965, -0.07834581, 0.03741648, -0.064613, -0.0044922354, 0.05150282, -0.01043456, -0.038883932, 0.0013263221, -0.047345605, -0.03669518, -0.073673025, 0.055956032, -0.0031711988, -0.0107596265, -0.053912353, -0.012321016, 0.0052027903, -0.05931239, -0.01181956, 0.03305743, -0.040960968, 0.03575864, -0.05874545, 0.038328175, -0.028642122, 0.1001079, -0.03132359, -0.061485533, 0.012711688, -0.04792668, -0.042671476, 0.021871619, -0.013868987, -0.097119875, 4.6119967E-4, 0.0038548494, -0.030143088, 0.04376323, -0.04252116, -0.003311639, -0.017443746, -0.012340171, -0.032708775, 0.035120785, -0.011805166, -0.01372578, -0.01727542, -0.0023261674, -0.040456187, 0.073783614, -0.041797835, -0.008352981, -0.094251916, 0.04974681, 0.03162338, 0.027761696, 0.03158548, -0.042769466, 0.002413028, -0.0028053017, 0.027332032, -0.013260667, -3.3364186E-4, 0.022478772, 0.018060198, 0.010788596, 0.009725766, 0.01791517, -0.017308813, -0.06276063, -0.022030298, -0.06577266, -0.006024345, 0.04061552, 0.06734963, 0.0169959, 0.03508686, -0.004924476, -0.007150875, 0.004072924, -0.02452725, 0.0026079526, 0.040827397, 0.02485745, 0.012311198, 0.04595321, 0.023110976, -0.008308107, -0.01349351, -0.0027289912, -0.020215554, 0.029206075, 0.008211485, -0.0020285, -0.007962775, -0.0011311583, 0.052420493, 0.029335976, -0.034268316, 0.027089413, -0.0141544575, -0.037790395, -0.032876235, -0.018181248, 0.051618215, -0.02391149, 0.040739205, 0.022332313, -0.022192521, -0.0038999524, 2.665285E-4, -0.026159858, -0.025313236, -0.011715555, -0.013942653, 0.009715638, -0.06985348, -0.03986098, 0.0470792, 0.0131037645, 0.0569468, -0.034224372, 0.019203423, -0.0026869678, -0.017593842, 0.023995357, -0.009984715, 0.010128069, 0.03743259, -0.019950591, 0.023682652, -0.014933193, 0.0012517767, -0.022865748, -0.001210017, 0.0015929559, -0.02579681, -0.0017350283, 0.018419998, -0.03629099, 0.0019098803, 0.020044416, 5.909709E-4, 0.019216752, -0.024207087, -0.0041541145, 0.014369621, 0.0037044152, -0.015547903, 0.04410028, 0.039407343, 0.0063609094, 0.018452296, 0.010365752, 0.0038212172, -0.017657751, -0.013848915, -0.03659765, 0.01227015, 0.032493267, 0.034269303, -0.020378161, -0.028054774, -0.0023942995, -0.006439075, -0.0050714565, 0.019598685, -0.024586435, -0.023228964, -0.008255578, -0.022321187, -0.010975467, -0.013256906, -0.032204706, 0.052073203, 0.0054438584, -0.0017145255, 0.02141263, -0.013922718, -0.0385722, 0.015181433, 0.022171939, -0.0053502037, -0.028801499, -0.012135697, -0.022278443, -0.017694848, -0.012493399, 0.0019240456, -0.025850736, -0.0060191327, 0.03563002, 0.0033583178, 0.015258327, -0.0051530795, 0.002398866, 0.020764885, 0.020225616, -0.0060330224, 0.029354535, -0.009926263, -0.01331316, 0.0076695136, -0.009092792, -0.016557172, 0.009162733, -0.008613232, 0.02971107, -0.017906457, -6.6019327E-4, -0.021018507, 0.0013300253, 0.001467901, 0.004009506, 0.010682631, 0.014998421, 0.0375694, -0.013299593, -0.027290402, -0.0019929102, -0.026408222, 0.0068023177, -0.02077405, -0.0073583247, 0.002652146, 0.045631643, 0.0046209916, -0.030087095, 0.003158622, -0.0012270092, 0.007513162, 0.015369132, 0.0061028395, -0.0050769504, 0.014694878, 0.016838461, -0.002496522, -0.003751302, 0.026715659, -0.01169101, -0.00583926, -0.0021141928, -0.00827598, 0.003451413, 0.0031923712, 0.01344758, 0.00936066, -0.0023082562, -0.0030353786, -0.0055100536, 0.03127455, 0.014757174, 0.024522578, 0.011656378, -0.017068626, 0.014585248, -0.021363627, -0.008750968, 0.013437441, 0.008366836, 0.0120168, -0.011006667, -0.0028149113, -0.0076802806, 0.021877414, 0.01408847, 0.035885923, -0.0072402735, -0.008659223, 0.009754819, 0.008619722, -0.0031400977, -0.026133742, -0.004457135, 0.018737331, 0.004529896, -0.010196676, 0.019900426, 0.015067387, 0.03402317, 0.01999167, -0.005758707, 0.00979926, -0.005611558, -0.0127474945, 0.014552781, -0.016785823, 0.009479103, 0.02297833, -0.010731883, -0.016717015, -0.017484287, 0.020361323, -0.013998314, 0.017850846, 0.014292788, 0.048527926, 0.010552317, -0.018746307, -0.017717497, 0.012798664, 0.0037520204, 0.0064832903, 0.024817422, -0.026294582, 0.0051326375, -0.0074774167, -0.0074702553, 0.0017218001, 8.0103375E-4, 0.005199527, 0.017515521, -0.0066512986, -0.0013254563, 0.0025569454, -0.010042196, -0.019547828, 0.00667863, 0.004732076, -0.005076369, -0.0051990394, 0.005639958, -0.008840068, -0.00202057, 0.004196233, 0.012377474, 0.011439983, -0.008849425, 0.01082493, -0.007176978, 0.0027949852, 0.001586264, -0.020166483, -0.019787777, -0.0276374, 0.016102798, -0.018139778, 0.009249267, 0.013843168, -0.004218314, -0.014982909, 0.017630966, 0.0018553302, -0.010762763, 0.010247762, 0.020129021, -0.02226605, 0.017677778, -0.0136600705, -0.021495359, 9.576419E-4, -0.0026726376, 0.0026236682, 0.009962605, -0.018459925, -0.021139728, 0.007942952, -0.01841508, 0.016816823, -0.0120083615, -0.020876758, 0.017487485, -0.011791492, 0.023464072, 1.6855022E-4, -0.019260358, -0.015247996, -0.020723242, -0.0072092214, 0.0038836407, -0.014971971, -0.0067802165, 0.012383788, 0.003148533, -0.001996739, 0.017085847, 0.01043297, 0.015849201, 0.022640757, 0.017766962, -0.02684469, 0.02430105, 0.011285774, 0.0044941027, 0.016542362, 0.009292583, 0.01089254, 0.010973071, 0.009921301, 0.013253412, 0.018811962, 0.008307326, 0.004228023, 0.008613386, -0.002340278, 0.019086504, 0.002332518, -0.012299346, 3.020229E-5, -0.007913956, -0.003973171, -0.017028747, 0.009567644, 0.008149756, 0.012768958, 0.001068468, 0.009620058, -0.004871017, -0.003550283, 0.012100322, -0.015033302, -0.008209837, 0.002292909, 0.023443846, 0.011615485, 0.0047445227, -9.560393E-4, -3.27645E-4, 0.00909202, 0.021312146, 0.003922159, -0.020681487, -0.013827459, -0.0049597328, 0.005719597, 0.003362033, 0.010312935, 0.0023698234, -0.012395746, -0.009986458, -0.013381608, 0.009090864, -0.010249262, 1.2023547E-5, 0.0038468342, 0.006337594, 0.0012656534, -0.021973915, 0.013570165, -0.002527013, 0.0025838292, -0.01267369, -0.01185739, -0.0012393842, -0.007108769, 0.019191096, 0.0076847733, -8.00497E-5, 0.0062433304, -0.0059921932, -0.007990699, 0.0028100375, -0.0067075496, -0.016492415, -0.01980759, -0.015540144, -0.0023266955, 0.0073061595, -9.437309E-4, -0.0027223274, -0.0017387761, -0.01786714, 0.016558483, 0.004781822, 0.013613918, 0.005038345, 0.0025672123, -0.004939192, 0.00478557, -0.003682138, -0.017883932, -0.0085733775, -0.0139945075, -0.023469679, -0.009169903, -0.003148576, -0.005561104, -0.008181999, 0.00439613, 0.015162514, 0.0063574747, 0.016413234, -0.004096887, -0.012458653, -0.018151276, -0.004095136, 0.00926797, -0.006555397, 0.008840552, 0.008436288, -0.0044730115, -0.017382707, -0.0025450094, -0.009555567, 0.0025377574, 0.011402533, -0.01082082, 0.0048879785, 0.008872678, 0.0071776286, -0.0028551456, 0.025974488, -0.0031310457, -0.021706022, -0.009738637, -6.46065E-4, -7.9623424E-4, -0.008404216, 0.003505369, -6.689662E-4, -0.016984435, -0.0036331743, 0.0063550733, -0.004627766, -0.0028516732, 0.006443027, 0.0020748319, 0.021670623, -0.0057692537, -0.003918615, -0.0031876706, -0.0027631929, -0.009873856, 0.0024800557, -0.008688225, -0.0015063204, -0.017516257, -0.021662358, 0.0040566046, 0.001759627, -0.00873719, 0.006285713, 0.014531575, -0.011670351, 9.434051E-4, -0.00920805, 0.01510458, -0.006492559, 0.011056852, 0.0018427926, -3.124221E-5, -0.015957613, 0.0021030544, -0.008498496, -0.017197903, -0.01116062, 0.00774098, 0.0061622197, -0.008144267, 0.009911781, 0.01610726, -0.0066612447, 0.0033935746, -0.014372329, 0.0045597753, -0.020207006, 0.0066753887, 0.021799188, 0.0040517636, 0.0045109354, 0.011718117, -2.6437384E-4, -5.938372E-4, 0.002140971, -0.0017630563, 0.0048767035, 0.017311148, -0.00504668, -0.0147418175, 0.0139642665, 0.001257399, 0.008115203, -0.0028234539, 0.005615775, 0.0020506762, -2.9385765E-4, -0.011488344, -4.8777653E-4, 0.005667841, 0.009290322, -0.012496263, -0.0025094815, -0.0024429646, 0.0090868, 0.003280879, -0.0019263525, 0.0030596212, 0.00457934, 0.007970626, 0.009429205, 0.0039575854, 0.0011453362, 0.003737679, 0.0062191375, -0.0033365435, 0.0055586784, 0.0046217036, -0.0059943916, 0.0101486705, 0.012657837, 4.8796192E-4, 0.005385128, 0.0041495743, -0.010394916, -0.007758405, -0.005766732, 0.011718328, -0.01692151, 9.872961E-4, -0.002198138, 0.016635805, 0.0027692034, -0.0050674435, 0.0011226385, -0.0021900255, -9.7107905E-4, -0.005269831, -0.007580122, 0.0032111898, 0.0020253612, -0.0044059856, -0.0037788139, 0.008817335, -0.0062701693, 0.005308133, -0.0013404146, -0.0061498955, 0.0059099253, -4.5824097E-4, -0.0046645263, -0.0020703068, 0.0010247212, -8.9980697E-4, 0.014734301, 0.008145946, 0.001952657, 4.9267797E-4, 4.033092E-4, 0.0016065567, -0.005226729, 0.010558401, -0.0063024457, 0.0018141957, 0.005426263, 0.009011342, -0.0055915676, 0.0029521552, -0.0030315302, 0.013015645, -0.003965996, -0.013479809, 0.011376977, -0.005542539, -0.012652742, -0.001385992, -0.018611869, 0.010869142, 0.0016872283, 0.0014726688, 0.008198611, -0.0077167833, -0.0104016615, 0.0032530972, 0.004940237, -0.005863709, 0.012621446, 0.0061782463, -0.012538319, 9.4925676E-4, -0.008842992, -0.006489274, -7.138018E-4, 0.007559196, 0.0016362873, 0.008001425, 0.0041235443, -0.0012436371, 0.011159509, 9.762285E-4, 0.009429023, -2.3953077E-4, 0.009857336, 0.0067546843, 0.004515829], vectorTwo=[-0.11973515, 0.08095747, -7.9745345E-4, 0.017778162, 0.03781757, 0.0948174, -0.010377076, -0.03181369, 0.05628834, 0.05891907, -0.030822765, 0.10757859, -0.23740931, -0.22470808, 0.1611373, -0.13324434, -0.10559932, 0.09629354, -0.18414927, -0.10610022, 0.108907506, -0.014238788, 0.13708599, -0.0932529, -0.046637416, -0.0723548, 0.021216057, -0.048501376, 0.10892839, -0.0034054436, 0.12404433, 0.11910919, 0.0407715, 0.030051203, 0.003348333, 0.048397772, 0.038554996, -0.006637996, 0.02692062, -0.05859029, 0.03757299, 0.06997503, 0.0697287, -0.016306043, 0.050575297, 0.035764903, -0.010194664, -0.03432703, -0.036454998, -0.040881, -0.0017276381, 0.119421884, -0.077626854, -0.01929169, -0.019275935, 0.17408307, 0.04608272, -0.02536122, 0.048946682, 0.026566505, 0.12054666, -0.014961985, 0.012728676, -0.021531789, -0.05905329, -0.06944409, -0.016700277, -0.060913816, -0.019436076, 0.08982007, 0.10389332, -0.102554955, 0.06951776, 0.07058402, -0.008742106, -0.02447418, 0.018724779, 0.028184239, 0.026121942, 0.013996168, 0.0664105, -0.038234126, -0.02303688, 0.04389385, 0.05041244, -0.11171202, 0.086676404, 0.028816389, -0.029978214, 0.043524325, -0.065447174, -0.07619985, -0.018115913, -0.066899836, -0.040868245, -0.10142577, 0.052631214, 0.039323542, -0.09457685, 0.027902136, -0.003998986, -0.037779003, 0.051883016, 3.1396424E-4, -0.03186952, -0.029589364, -0.024453204, 0.009792744, 0.002296945, -0.04543461, -0.034005456, 0.043245226, 0.038796358, -0.07531766, -0.034181666, -0.069799244, 0.001988578, 0.051464427, -0.09935535, -0.030321337, 0.010854493, 0.052270748, 0.041293442, -0.11673548, -0.034175783, 0.061309226, -0.06036236, 0.07255635, 0.009921444, -0.035960246, -0.047523238, -0.009834568, -0.040689506, -0.018173918, 0.011758854, 0.06473035, -0.023727052, 0.01765984, -0.0059596286, 0.023125466, -0.03085419, -0.018143924, 0.09201374, 0.009294985, -0.03208583, 0.012164934, 0.07489599, -0.06952085, -0.055458255, -0.04811663, 0.02467739, 0.014218112, 0.010280043, 4.3671473E-4, -0.0394551, 0.041456345, 0.0013710422, -0.009683716, 0.018802414, -0.006729811, -0.009802281, 0.018261077, 0.052742865, -0.022915887, 0.06353675, -0.007284017, -0.002082828, 0.007783691, -0.03130499, 0.023800777, -0.02610662, -0.023864703, 0.0068920017, 0.019401008, -0.0056829136, -5.0605834E-4, -0.053986073, -0.065493464, -0.034092687, 0.04220041, 0.009476794, 0.050892465, 0.005893973, -0.027504334, 0.0028201283, 0.0027392025, 0.027911827, 0.018575583, -0.005915392, 0.026672399, -0.009604471, -0.02643193, 0.0013825679, 0.061390538, 0.08235732, 0.031154502, 0.037601363, -0.016688455, 0.038007755, 0.07273599, 0.014851937, -0.0076043806, -0.07303425, -0.03740918, 0.019913351, 2.030503E-4, 0.0068934127, -0.0034623286, -0.024364877, 0.020607062, -0.035610355, 0.003314935, -0.044780705, -0.05102975, 0.026802188, 0.013005619, -0.012983566, -0.05817688, 0.012086601, -0.07851603, -0.032840643, 0.015080899, 0.0048408033, -0.022132527, -0.043668665, -0.024322625, 0.024781957, -0.026795862, 0.017768756, 0.0576815, -0.031258337, -0.04034984, -0.0022927672, 0.06328568, -0.05691621, -0.029289957, 0.007887512, -0.025037454, 0.052291643, -0.0042550424, -0.002641092, -0.03615621, -0.00850641, 0.00269375, 0.009949909, -0.011824053, 0.003560511, 0.055153664, 0.026184153, -0.06295364, -0.0066079907, -0.014661856, 0.035361435, 0.04479921, -0.01436901, -0.025124198, -0.023145422, -0.022270745, 0.013288247, -0.02767609, -0.01146416, 0.0145266075, -0.0054158275, -0.010314086, 0.009966933, 0.01918325, -0.02135528, 0.022194697, 0.014061586, 0.021323303, -0.0012787688, -0.011617605, -0.044630684, -0.0014462662, -0.055514146, 0.013914377, -0.012164333, 0.012878909, 0.009302253, -0.037286866, 0.025333503, -0.04553513, -0.016631879, -0.027500989, -0.047244523, 0.030365756, -0.012928509, -0.056037493, 0.002111567, -3.445983E-4, 0.01656157, 0.023865225, 0.014607268, -0.03616821, 0.02551308, -0.009524459, -0.031222345, -0.02033182, 0.057622015, 0.019413684, -0.015354169, 2.039349E-4, 0.0052822824, -0.02027396, 0.0038262396, -0.032031342, -0.005577524, -0.006228494, 0.013997069, -0.026569499, -0.007623818, -0.026430082, -0.037221637, -0.0011455176, 0.026016934, 0.0051541305, -0.04175314, -0.024648841, -0.03386957, 0.024139646, -0.027392618, -0.028452821, -0.0047744913, 0.026683511, -0.043278668, -0.008025299, 0.0062434734, 0.023799315, 0.018116387, -0.017212955, 0.00675846, 0.017144578, -0.011222329, -0.047040876, 0.011713254, 0.030690365, 0.020916743, 0.024264934, 0.0026452346, -0.031266063, 0.018595783, 0.008722972, 0.0011967237, 0.0240953, 0.007855105, -0.027829738, -0.0018804515, 0.0045171697, 0.0012034165, -0.025808694, -0.031137438, 0.012954249, -0.02222534, 0.007469989, -0.03299767, 0.019790541, 0.019729232, 0.025672013, 0.00655929, 0.019496253, 0.005682889, 0.026479306, 0.0050754775, -0.0039409203, 0.06357067, -0.028487764, -0.03659115, -0.02870191, 0.00958703, -0.0073981844, 0.0051872428, 0.0025473204, 0.006272702, -0.002030124, -0.028699424, -0.010338337, -0.023391975, -0.010535413, 0.012972192, 0.030313427, -0.017857233, -0.02584283, -0.026900806, -0.0029144983, -0.005277483, 0.014036093, -0.019638328, 0.029866232, -0.013569687, -0.025121475, -0.027261615, 0.009427129, 0.06471093, 0.015230324, -0.0056302394, -0.011471548, 0.017689008, 0.004637535, 0.0011149658, -0.031982716, 0.022016572, -0.0110083725, -0.0042767446, -6.200085E-4, 0.056941316, 0.01131996, -0.0036733507, -0.009270783, -0.010277945, -0.03191185, -0.0146463355, -0.005157946, -0.029410489, 0.03908128, 0.036739152, -0.017181905, -0.017025935, -0.009147714, 0.028543074, -0.027474517, 0.012574309, -0.005393016, 0.017847693, 0.01726672, 0.010323364, -0.0135736335, 0.0069461744, -0.022242447, 0.0058118105, 0.020939272, 0.0045747184, 0.014999139, 0.003252502, -0.016419375, 0.031438574, 0.04617525, 0.00784853, -0.0012723412, -0.025017604, 0.029681016, -0.024697304, -0.0052957274, 0.009767601, 0.0051521296, -0.0012994702, -0.0017888923, 0.044997703, -0.02585113, 0.020166703, -0.026824266, 0.025578074, 0.021925531, 0.014210808, 0.039668396, -0.005146907, 0.002887164, -0.003076182, -0.006285401, -0.023103757, -0.030286463, -0.027537484, 0.0030390045, -0.013392114, -0.0031120274, -0.018732185, 2.4149122E-4, 0.025475236, -0.029740073, -0.0056005623, -0.026741868, 0.0055567245, -0.008290855, 0.006020938, 0.017970854, 0.003936837, 0.011447729, 0.011691844, -0.0012378974, -0.0013880703, 0.036667068, 0.0128294565, -0.001602585, -0.0053273225, -0.013628352, -0.020104848, -0.0025651231, 0.0027692064, 0.041127414, -0.03252928, 0.015430981, 0.014353925, -0.035179876, 0.004730329, 0.0032820883, 0.019320745, 0.0070202886, -0.040679436, -0.034837708, -0.008630759, 0.009457761, 0.004813033, -0.020194704, -0.0034597763, -0.006086866, -0.012982794, -0.012567925, 0.013816319, -0.018442733, -0.018542282, -0.017392524, 0.007060914, 0.005591555, 0.0038210428, 0.020253645, -0.022122532, -0.010383561, 0.0068959575, -0.020679463, 0.028952265, -0.0073548774, 0.011456289, 1.1035489E-4, 6.2163285E-4, -0.014207815, 0.011708225, 0.009786873, -0.020304127, 0.019790946, -0.004707331, -0.0043455944, -0.001971539, 0.021671792, 3.45811E-4, -0.009738243, -0.014583063, -0.023843648, -0.017617645, -0.0073720664, -0.029533107, -0.0017647085, 0.02123342, 0.015121319, -0.024342237, 0.0062186574, 0.009744628, 0.032493576, 0.019315507, 0.014841595, 0.006602987, -6.539175E-4, -0.012376351, 0.026424857, -0.011808081, 0.024625171, 0.054046284, 0.004188485, -0.0016174468, 0.017886845, -0.012949441, 0.027347645, 0.021280972, -8.869145E-4, -0.019538047, -0.01612249, -0.015336986, -0.01007112, -0.001580681, 1.2961272E-4, -0.009443218, -0.013082444, 0.0037698112, -0.0049349843, -0.015419478, -0.028887676, -0.015861291, -0.02972866, -0.025518926, 0.0011080995, 0.01872132, -0.007845809, -0.01142388, 0.0010394778, 0.0076191085, -0.03206946, 0.0028279186, 0.0252884, -0.015494946, -0.00487451, -0.03042514, 0.0056792204, 0.00557942, 0.01477863, -0.016854461, -0.016603416, 0.033179272, 0.018492555, -0.018943263, -0.019813849, -0.02169808, 0.026190136, -0.0026397225, 0.012093653, -0.020144835, -0.003413523, -0.019216709, -0.016063567, 0.01053034, -0.019057969, 0.009921677, 0.017265338, -0.008042013, -0.032106385, -0.0076573566, -0.011284133, 0.0016440647, -0.0074232114, -0.0061121606, -0.0039731245, 0.017772922, -0.006773471, -0.003257682, -0.014163464, -0.0053975377, -0.03454502, -0.005380314, 0.025113512, -0.037035916, 0.005860556, 0.009815752, 0.0013864046, -0.021046035, -0.010914981, 0.02522609, 0.003725368, 0.024608297, -0.008895582, 0.020630056, -0.02082611, 0.005259784, -0.004662828, 0.018079981, -0.018179085, -0.012555717, -0.03287497, 0.024182148, 9.2749647E-4, -0.002516815, 0.0031977864, 0.0034360823, 0.022472538, -0.0093675945, 0.006006379, 0.007345346, 0.005745967, 0.008020854, 0.018388424, 4.7079913E-4, 0.0043508816, 0.0032794503, -0.008247611, 6.538056E-4, 0.006181467, 0.026531989, -0.009177468, -0.01658023, -0.0015889334, 0.022452582, 0.035810877, 0.013010471, 0.0015583939, -0.018390406, 0.0031738477, -0.021265933, -0.008688021, -0.009019973, 0.0069699707, 0.005563408, 0.009978513, -0.005430095, -0.009088654, -0.00735871, -0.0022795217, 0.01065763, -0.019584652, 0.0039610327, -0.01614338, -0.014655342, -0.009314964, -0.009561065, -0.022128796, 0.0051152497, 7.2398904E-4, 0.011731552, 0.019381693, -0.0018754478, 0.015200723, 0.01111195, -0.0057720356, 0.011230107, -0.0033303176, -0.0077977283, -0.012010337, 0.009293151, 2.3450867E-4, 0.018484943, 0.0050424044, -0.002583624, 0.003367546, 0.0018279159, -0.0017556995, 0.010968927, 0.009355919, 0.024280738, -0.0028832199, 0.019350704, 0.0015977627, 0.003447935, 0.017883845, 0.014826211, -0.02617111, -0.024304014, -0.0041993647, -0.020207208, -0.008573829, 0.0033712676, -0.006179999, 0.007474309, 0.0029519126, -0.0014506467, -0.013240714, -0.016551293, 0.012099103, -0.0050255316, -0.0057253223, -0.0021310174, -0.008341446, 0.0041917237, -0.008869906, 0.011595749, -0.0016620221, 0.007877958, 0.008671747, 0.016892435, 0.008028487, -0.02293363, 0.0087649925, -0.03605261, -0.013018509, -0.011498792, 0.010009394, 0.010596368, -0.0097576715, 0.007993852, 0.010651219, 0.0043800627, -9.224701E-4, 0.040351115, -0.0011116522, -0.02665772, 0.013831533, 0.0020280543, 0.0116251325, -0.032511443, -0.0046799295, 0.008501904, 0.0048212362, 0.0026921472, -0.0062492155]}}}",
                                "time_in_nanos": 654798645,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1497602,
                                    "match": 0,
                                    "score_count": 62720,
                                    "next_doc_count": 62739,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "score": 653147513,
                                    "count_weight_count": 0,
                                    "build_scorer_count": 38,
                                    "create_weight": 6050,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 147480
                                }
                            }
                        ],
                        "rewrite_time": 9400,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 657672371,
                                "children": [
                                    {
                                        "name": "TopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 654599242
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 899170,
                    "breakdown": {
                        "load_stored_fields": 532480,
                        "load_source": 5740,
                        "load_stored_fields_count": 4,
                        "next_reader_count": 2,
                        "load_source_count": 4,
                        "next_reader": 84640
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 59690,
                            "breakdown": {
                                "process_count": 4,
                                "process": 28760,
                                "next_reader": 30930,
                                "next_reader_count": 2
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 15550,
                            "breakdown": {
                                "process_count": 4,
                                "process": 15360,
                                "next_reader": 190,
                                "next_reader_count": 2
                            },
                            "debug": {
                                "fast_path": 4
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1620,
                            "breakdown": {
                                "process_count": 4,
                                "process": 1260,
                                "next_reader": 360,
                                "next_reader_count": 2
                            }
                        }
                    ]
                }
            },
            {
                "id": "[C6ztsNfJS56-Nbzhk-Co1Q][vectest][1]",
                "node_id": "C6ztsNfJS56-Nbzhk-Co1Q",
                "shard_id": 1,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 653619932,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1510096,
                                    "match": 0,
                                    "score_count": 62345,
                                    "next_doc_count": 62362,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "score": 651918076,
                                    "count_weight_count": 0,
                                    "build_scorer_count": 34,
                                    "create_weight": 6050,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 185710
                                }
                            }
                        ],
                        "rewrite_time": 9250,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 656181861,
                                "children": [
                                    {
                                        "name": "TopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 653138762
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 706620,
                    "breakdown": {
                        "load_stored_fields": 454340,
                        "load_source": 3080,
                        "load_stored_fields_count": 3,
                        "next_reader_count": 3,
                        "load_source_count": 3,
                        "next_reader": 59130
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 47090,
                            "breakdown": {
                                "process_count": 3,
                                "process": 25600,
                                "next_reader": 21490,
                                "next_reader_count": 3
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 14560,
                            "breakdown": {
                                "process_count": 3,
                                "process": 14290,
                                "next_reader": 270,
                                "next_reader_count": 3
                            },
                            "debug": {
                                "fast_path": 3
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1610,
                            "breakdown": {
                                "process_count": 3,
                                "process": 1180,
                                "next_reader": 430,
                                "next_reader_count": 3
                            }
                        }
                    ]
                }
            },
            {
                "id": "[C6ztsNfJS56-Nbzhk-Co1Q][vectest][2]",
                "node_id": "C6ztsNfJS56-Nbzhk-Co1Q",
                "shard_id": 2,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 645363978,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1477267,
                                    "match": 0,
                                    "score_count": 62480,
                                    "next_doc_count": 62497,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "score": 643771321,
                                    "count_weight_count": 0,
                                    "build_scorer_count": 34,
                                    "create_weight": 2050,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 113340
                                }
                            }
                        ],
                        "rewrite_time": 10200,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 648581635,
                                "children": [
                                    {
                                        "name": "TopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 645550526
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": []
            },
            {
                "id": "[C6ztsNfJS56-Nbzhk-Co1Q][vectest][3]",
                "node_id": "C6ztsNfJS56-Nbzhk-Co1Q",
                "shard_id": 3,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 642892573,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1589522,
                                    "match": 0,
                                    "score_count": 62448,
                                    "next_doc_count": 62467,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "score": 641043642,
                                    "count_weight_count": 0,
                                    "build_scorer_count": 38,
                                    "create_weight": 6180,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 253229
                                }
                            }
                        ],
                        "rewrite_time": 9220,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 645238087,
                                "children": [
                                    {
                                        "name": "TopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 642178837
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 1398869,
                    "breakdown": {
                        "load_stored_fields": 1039580,
                        "load_source": 2430,
                        "load_stored_fields_count": 3,
                        "next_reader_count": 3,
                        "load_source_count": 3,
                        "next_reader": 86470
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 54690,
                            "breakdown": {
                                "process_count": 3,
                                "process": 15040,
                                "next_reader": 39650,
                                "next_reader_count": 3
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 8930,
                            "breakdown": {
                                "process_count": 3,
                                "process": 8650,
                                "next_reader": 280,
                                "next_reader_count": 3
                            },
                            "debug": {
                                "fast_path": 3
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1270,
                            "breakdown": {
                                "process_count": 3,
                                "process": 870,
                                "next_reader": 400,
                                "next_reader_count": 3
                            }
                        }
                    ]
                }
            }
        ]
    }

Here’s the 9.1.3 JVM info:

{
                "pid": 175067,
                "version": "24.0.2",
                "vm_name": "OpenJDK 64-Bit Server VM",
                "vm_version": "24.0.2+12-54",
                "vm_vendor": "Oracle Corporation",
                "using_bundled_jdk": true,
                "start_time_in_millis": 1757701020749,
                "mem": {
                    "heap_init_in_bytes": 4294967296,
                    "heap_max_in_bytes": 4294967296,
                    "non_heap_init_in_bytes": 7667712,
                    "non_heap_max_in_bytes": 0,
                    "direct_max_in_bytes": 2147483648
                },
                "gc_collectors": [
                    "G1 Young Generation",
                    "G1 Concurrent GC",
                    "G1 Old Generation"
                ],
                "memory_pools": [
                    "CodeHeap 'non-nmethods'",
                    "Metaspace",
                    "CodeHeap 'profiled nmethods'",
                    "Compressed Class Space",
                    "G1 Eden Space",
                    "G1 Old Gen",
                    "G1 Survivor Space",
                    "CodeHeap 'non-profiled nmethods'"
                ],
                "using_compressed_ordinary_object_pointers": "true",
                "input_arguments": [
                    "-Des.networkaddress.cache.ttl=60",
                    "-Des.networkaddress.cache.negative.ttl=10",
                    "-XX:+AlwaysPreTouch",
                    "-Xss1m",
                    "-Djava.awt.headless=true",
                    "-Dfile.encoding=UTF-8",
                    "-Djna.nosys=true",
                    "-XX:-OmitStackTraceInFastThrow",
                    "-Dio.netty.noUnsafe=true",
                    "-Dio.netty.noKeySetOptimization=true",
                    "-Dio.netty.recycler.maxCapacityPerThread=0",
                    "--add-opens=org.apache.lucene.core/org.apache.lucene.codecs.lucene99=org.elasticsearch.server",
                    "--add-opens=org.apache.lucene.backward_codecs/org.apache.lucene.backward_codecs.lucene90=org.elasticsearch.server",
                    "--add-opens=org.apache.lucene.backward_codecs/org.apache.lucene.backward_codecs.lucene91=org.elasticsearch.server",
                    "--add-opens=org.apache.lucene.backward_codecs/org.apache.lucene.backward_codecs.lucene92=org.elasticsearch.server",
                    "--add-opens=org.apache.lucene.backward_codecs/org.apache.lucene.backward_codecs.lucene94=org.elasticsearch.server",
                    "--add-opens=org.apache.lucene.backward_codecs/org.apache.lucene.backward_codecs.lucene95=org.elasticsearch.server",
                    "-Dlog4j.shutdownHookEnabled=false",
                    "-Dlog4j2.disable.jmx=true",
                    "-Dlog4j2.formatMsgNoLookups=true",
                    "-Djava.locale.providers=CLDR",
                    "-Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24",
                    "-Des.path.home=/usr/share/elasticsearch",
                    "-Des.distribution.type=deb",
                    "-Des.java.type=bundled JDK",
                    "--enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core",
                    "--enable-native-access=ALL-UNNAMED",
                    "--illegal-native-access=deny",
                    "-XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log",
                    "-Des.entitlements.enabled=true",
                    "-XX:+EnableDynamicAgentLoading",
                    "-Djdk.attach.allowAttachSelf=true",
                    "--patch-module=java.base=/usr/share/elasticsearch/lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.1.3.jar",
                    "--add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net",
                    "-Xms4g",
                    "-Xmx4g",
                    "-XX:+UseG1GC",
                    "-Djava.io.tmpdir=/tmp/elasticsearch-7818474812430334929",
                    "--add-modules=jdk.incubator.vector",
                    "-Dorg.apache.lucene.store.defaultReadAdvice=normal",
                    "-Dvector.rescoring.directio=false",
                    "-XX:+HeapDumpOnOutOfMemoryError",
                    "-XX:+ExitOnOutOfMemoryError",
                    "-XX:ErrorFile=hs_err_pid%p.log",
                    "-Xlog:gc*,gc+age=trace,safepoint:file=gc.log:utctime,level,pid,tags:filecount=8,filesize=64m",
                    "-Dlog4j2.formatMsgNoLookups=true",
                    "-XX:MaxDirectMemorySize=2147483648",
                    "-XX:G1HeapRegionSize=4m",
                    "-XX:InitiatingHeapOccupancyPercent=30",
                    "-XX:G1ReservePercent=15",
                    "--module-path=/usr/share/elasticsearch/lib",
                    "--add-modules=jdk.net",
                    "--add-modules=jdk.management.agent",
                    "--add-modules=ALL-MODULE-PATH",
                    "-Djdk.module.main=org.elasticsearch.server"
                ]
            }

Profile from 8.19.3 for the same query:

{
        "shards": [
            {
                "id": "[9TeIad8DS7qgMWTXJgxOow][vectest][0]",
                "node_id": "9TeIad8DS7qgMWTXJgxOow",
                "shard_id": 0,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "script_score (*:*, script: {Script{type=inline, lang='painless', idOrCode='Math.max(cosineSimilarity(params.vectorOne, 'vectorOne') + cosineSimilarity(params.vectorTwo, 'vectorTwo'), 0)', options={}, params={vectorOne=[0.013173203, 0.0052297856, 0.21519354, -0.093842305, -0.01095355, -0.29436445, 0.06493544, 0.05938409, -0.17600165, -0.20461452, -0.015728287, 0.0046440605, -0.030841334, 0.122089006, -0.3034737, 0.037594393, -0.10002559, 0.11257306, -0.111819915, -0.12071488, 0.020595897, 0.15776865, 0.052580394, -0.09745094, 0.079588, -0.16879044, -0.034594744, -0.050949633, 0.08187252, -0.16822045, -0.052350275, 0.048628043, 0.07935129, 0.03799321, 0.10957204, 0.028511189, 0.17220527, -0.12096245, -0.066655464, 0.08931657, -0.054182395, 0.12907726, -0.03926433, 0.058102854, -0.17350669, 0.03771998, -0.08872518, 0.011460013, -0.01628334, 0.07906166, -0.05854619, -0.05143622, -0.09604408, -0.09826985, 0.045209, 0.05044051, 0.02117565, -0.057305027, 0.04241245, 0.043347206, 0.07159769, 0.018550502, 0.04424336, 0.06912926, -0.009729412, 0.03211654, 0.04782096, -0.08818417, -0.035109524, 0.018651119, 0.03159385, 0.022730356, -0.020502992, -0.066785574, -0.029437877, 0.018001571, -0.08194266, -0.056691643, 0.027324012, 0.025953125, 0.058490414, 0.031390805, -0.04929738, 0.02078965, -0.07834581, 0.03741648, -0.064613, -0.0044922354, 0.05150282, -0.01043456, -0.038883932, 0.0013263221, -0.047345605, -0.03669518, -0.073673025, 0.055956032, -0.0031711988, -0.0107596265, -0.053912353, -0.012321016, 0.0052027903, -0.05931239, -0.01181956, 0.03305743, -0.040960968, 0.03575864, -0.05874545, 0.038328175, -0.028642122, 0.1001079, -0.03132359, -0.061485533, 0.012711688, -0.04792668, -0.042671476, 0.021871619, -0.013868987, -0.097119875, 4.6119967E-4, 0.0038548494, -0.030143088, 0.04376323, -0.04252116, -0.003311639, -0.017443746, -0.012340171, -0.032708775, 0.035120785, -0.011805166, -0.01372578, -0.01727542, -0.0023261674, -0.040456187, 0.073783614, -0.041797835, -0.008352981, -0.094251916, 0.04974681, 0.03162338, 0.027761696, 0.03158548, -0.042769466, 0.002413028, -0.0028053017, 0.027332032, -0.013260667, -3.3364186E-4, 0.022478772, 0.018060198, 0.010788596, 0.009725766, 0.01791517, -0.017308813, -0.06276063, -0.022030298, -0.06577266, -0.006024345, 0.04061552, 0.06734963, 0.0169959, 0.03508686, -0.004924476, -0.007150875, 0.004072924, -0.02452725, 0.0026079526, 0.040827397, 0.02485745, 0.012311198, 0.04595321, 0.023110976, -0.008308107, -0.01349351, -0.0027289912, -0.020215554, 0.029206075, 0.008211485, -0.0020285, -0.007962775, -0.0011311583, 0.052420493, 0.029335976, -0.034268316, 0.027089413, -0.0141544575, -0.037790395, -0.032876235, -0.018181248, 0.051618215, -0.02391149, 0.040739205, 0.022332313, -0.022192521, -0.0038999524, 2.665285E-4, -0.026159858, -0.025313236, -0.011715555, -0.013942653, 0.009715638, -0.06985348, -0.03986098, 0.0470792, 0.0131037645, 0.0569468, -0.034224372, 0.019203423, -0.0026869678, -0.017593842, 0.023995357, -0.009984715, 0.010128069, 0.03743259, -0.019950591, 0.023682652, -0.014933193, 0.0012517767, -0.022865748, -0.001210017, 0.0015929559, -0.02579681, -0.0017350283, 0.018419998, -0.03629099, 0.0019098803, 0.020044416, 5.909709E-4, 0.019216752, -0.024207087, -0.0041541145, 0.014369621, 0.0037044152, -0.015547903, 0.04410028, 0.039407343, 0.0063609094, 0.018452296, 0.010365752, 0.0038212172, -0.017657751, -0.013848915, -0.03659765, 0.01227015, 0.032493267, 0.034269303, -0.020378161, -0.028054774, -0.0023942995, -0.006439075, -0.0050714565, 0.019598685, -0.024586435, -0.023228964, -0.008255578, -0.022321187, -0.010975467, -0.013256906, -0.032204706, 0.052073203, 0.0054438584, -0.0017145255, 0.02141263, -0.013922718, -0.0385722, 0.015181433, 0.022171939, -0.0053502037, -0.028801499, -0.012135697, -0.022278443, -0.017694848, -0.012493399, 0.0019240456, -0.025850736, -0.0060191327, 0.03563002, 0.0033583178, 0.015258327, -0.0051530795, 0.002398866, 0.020764885, 0.020225616, -0.0060330224, 0.029354535, -0.009926263, -0.01331316, 0.0076695136, -0.009092792, -0.016557172, 0.009162733, -0.008613232, 0.02971107, -0.017906457, -6.6019327E-4, -0.021018507, 0.0013300253, 0.001467901, 0.004009506, 0.010682631, 0.014998421, 0.0375694, -0.013299593, -0.027290402, -0.0019929102, -0.026408222, 0.0068023177, -0.02077405, -0.0073583247, 0.002652146, 0.045631643, 0.0046209916, -0.030087095, 0.003158622, -0.0012270092, 0.007513162, 0.015369132, 0.0061028395, -0.0050769504, 0.014694878, 0.016838461, -0.002496522, -0.003751302, 0.026715659, -0.01169101, -0.00583926, -0.0021141928, -0.00827598, 0.003451413, 0.0031923712, 0.01344758, 0.00936066, -0.0023082562, -0.0030353786, -0.0055100536, 0.03127455, 0.014757174, 0.024522578, 0.011656378, -0.017068626, 0.014585248, -0.021363627, -0.008750968, 0.013437441, 0.008366836, 0.0120168, -0.011006667, -0.0028149113, -0.0076802806, 0.021877414, 0.01408847, 0.035885923, -0.0072402735, -0.008659223, 0.009754819, 0.008619722, -0.0031400977, -0.026133742, -0.004457135, 0.018737331, 0.004529896, -0.010196676, 0.019900426, 0.015067387, 0.03402317, 0.01999167, -0.005758707, 0.00979926, -0.005611558, -0.0127474945, 0.014552781, -0.016785823, 0.009479103, 0.02297833, -0.010731883, -0.016717015, -0.017484287, 0.020361323, -0.013998314, 0.017850846, 0.014292788, 0.048527926, 0.010552317, -0.018746307, -0.017717497, 0.012798664, 0.0037520204, 0.0064832903, 0.024817422, -0.026294582, 0.0051326375, -0.0074774167, -0.0074702553, 0.0017218001, 8.0103375E-4, 0.005199527, 0.017515521, -0.0066512986, -0.0013254563, 0.0025569454, -0.010042196, -0.019547828, 0.00667863, 0.004732076, -0.005076369, -0.0051990394, 0.005639958, -0.008840068, -0.00202057, 0.004196233, 0.012377474, 0.011439983, -0.008849425, 0.01082493, -0.007176978, 0.0027949852, 0.001586264, -0.020166483, -0.019787777, -0.0276374, 0.016102798, -0.018139778, 0.009249267, 0.013843168, -0.004218314, -0.014982909, 0.017630966, 0.0018553302, -0.010762763, 0.010247762, 0.020129021, -0.02226605, 0.017677778, -0.0136600705, -0.021495359, 9.576419E-4, -0.0026726376, 0.0026236682, 0.009962605, -0.018459925, -0.021139728, 0.007942952, -0.01841508, 0.016816823, -0.0120083615, -0.020876758, 0.017487485, -0.011791492, 0.023464072, 1.6855022E-4, -0.019260358, -0.015247996, -0.020723242, -0.0072092214, 0.0038836407, -0.014971971, -0.0067802165, 0.012383788, 0.003148533, -0.001996739, 0.017085847, 0.01043297, 0.015849201, 0.022640757, 0.017766962, -0.02684469, 0.02430105, 0.011285774, 0.0044941027, 0.016542362, 0.009292583, 0.01089254, 0.010973071, 0.009921301, 0.013253412, 0.018811962, 0.008307326, 0.004228023, 0.008613386, -0.002340278, 0.019086504, 0.002332518, -0.012299346, 3.020229E-5, -0.007913956, -0.003973171, -0.017028747, 0.009567644, 0.008149756, 0.012768958, 0.001068468, 0.009620058, -0.004871017, -0.003550283, 0.012100322, -0.015033302, -0.008209837, 0.002292909, 0.023443846, 0.011615485, 0.0047445227, -9.560393E-4, -3.27645E-4, 0.00909202, 0.021312146, 0.003922159, -0.020681487, -0.013827459, -0.0049597328, 0.005719597, 0.003362033, 0.010312935, 0.0023698234, -0.012395746, -0.009986458, -0.013381608, 0.009090864, -0.010249262, 1.2023547E-5, 0.0038468342, 0.006337594, 0.0012656534, -0.021973915, 0.013570165, -0.002527013, 0.0025838292, -0.01267369, -0.01185739, -0.0012393842, -0.007108769, 0.019191096, 0.0076847733, -8.00497E-5, 0.0062433304, -0.0059921932, -0.007990699, 0.0028100375, -0.0067075496, -0.016492415, -0.01980759, -0.015540144, -0.0023266955, 0.0073061595, -9.437309E-4, -0.0027223274, -0.0017387761, -0.01786714, 0.016558483, 0.004781822, 0.013613918, 0.005038345, 0.0025672123, -0.004939192, 0.00478557, -0.003682138, -0.017883932, -0.0085733775, -0.0139945075, -0.023469679, -0.009169903, -0.003148576, -0.005561104, -0.008181999, 0.00439613, 0.015162514, 0.0063574747, 0.016413234, -0.004096887, -0.012458653, -0.018151276, -0.004095136, 0.00926797, -0.006555397, 0.008840552, 0.008436288, -0.0044730115, -0.017382707, -0.0025450094, -0.009555567, 0.0025377574, 0.011402533, -0.01082082, 0.0048879785, 0.008872678, 0.0071776286, -0.0028551456, 0.025974488, -0.0031310457, -0.021706022, -0.009738637, -6.46065E-4, -7.9623424E-4, -0.008404216, 0.003505369, -6.689662E-4, -0.016984435, -0.0036331743, 0.0063550733, -0.004627766, -0.0028516732, 0.006443027, 0.0020748319, 0.021670623, -0.0057692537, -0.003918615, -0.0031876706, -0.0027631929, -0.009873856, 0.0024800557, -0.008688225, -0.0015063204, -0.017516257, -0.021662358, 0.0040566046, 0.001759627, -0.00873719, 0.006285713, 0.014531575, -0.011670351, 9.434051E-4, -0.00920805, 0.01510458, -0.006492559, 0.011056852, 0.0018427926, -3.124221E-5, -0.015957613, 0.0021030544, -0.008498496, -0.017197903, -0.01116062, 0.00774098, 0.0061622197, -0.008144267, 0.009911781, 0.01610726, -0.0066612447, 0.0033935746, -0.014372329, 0.0045597753, -0.020207006, 0.0066753887, 0.021799188, 0.0040517636, 0.0045109354, 0.011718117, -2.6437384E-4, -5.938372E-4, 0.002140971, -0.0017630563, 0.0048767035, 0.017311148, -0.00504668, -0.0147418175, 0.0139642665, 0.001257399, 0.008115203, -0.0028234539, 0.005615775, 0.0020506762, -2.9385765E-4, -0.011488344, -4.8777653E-4, 0.005667841, 0.009290322, -0.012496263, -0.0025094815, -0.0024429646, 0.0090868, 0.003280879, -0.0019263525, 0.0030596212, 0.00457934, 0.007970626, 0.009429205, 0.0039575854, 0.0011453362, 0.003737679, 0.0062191375, -0.0033365435, 0.0055586784, 0.0046217036, -0.0059943916, 0.0101486705, 0.012657837, 4.8796192E-4, 0.005385128, 0.0041495743, -0.010394916, -0.007758405, -0.005766732, 0.011718328, -0.01692151, 9.872961E-4, -0.002198138, 0.016635805, 0.0027692034, -0.0050674435, 0.0011226385, -0.0021900255, -9.7107905E-4, -0.005269831, -0.007580122, 0.0032111898, 0.0020253612, -0.0044059856, -0.0037788139, 0.008817335, -0.0062701693, 0.005308133, -0.0013404146, -0.0061498955, 0.0059099253, -4.5824097E-4, -0.0046645263, -0.0020703068, 0.0010247212, -8.9980697E-4, 0.014734301, 0.008145946, 0.001952657, 4.9267797E-4, 4.033092E-4, 0.0016065567, -0.005226729, 0.010558401, -0.0063024457, 0.0018141957, 0.005426263, 0.009011342, -0.0055915676, 0.0029521552, -0.0030315302, 0.013015645, -0.003965996, -0.013479809, 0.011376977, -0.005542539, -0.012652742, -0.001385992, -0.018611869, 0.010869142, 0.0016872283, 0.0014726688, 0.008198611, -0.0077167833, -0.0104016615, 0.0032530972, 0.004940237, -0.005863709, 0.012621446, 0.0061782463, -0.012538319, 9.4925676E-4, -0.008842992, -0.006489274, -7.138018E-4, 0.007559196, 0.0016362873, 0.008001425, 0.0041235443, -0.0012436371, 0.011159509, 9.762285E-4, 0.009429023, -2.3953077E-4, 0.009857336, 0.0067546843, 0.004515829], vectorTwo=[-0.11973515, 0.08095747, -7.9745345E-4, 0.017778162, 0.03781757, 0.0948174, -0.010377076, -0.03181369, 0.05628834, 0.05891907, -0.030822765, 0.10757859, -0.23740931, -0.22470808, 0.1611373, -0.13324434, -0.10559932, 0.09629354, -0.18414927, -0.10610022, 0.108907506, -0.014238788, 0.13708599, -0.0932529, -0.046637416, -0.0723548, 0.021216057, -0.048501376, 0.10892839, -0.0034054436, 0.12404433, 0.11910919, 0.0407715, 0.030051203, 0.003348333, 0.048397772, 0.038554996, -0.006637996, 0.02692062, -0.05859029, 0.03757299, 0.06997503, 0.0697287, -0.016306043, 0.050575297, 0.035764903, -0.010194664, -0.03432703, -0.036454998, -0.040881, -0.0017276381, 0.119421884, -0.077626854, -0.01929169, -0.019275935, 0.17408307, 0.04608272, -0.02536122, 0.048946682, 0.026566505, 0.12054666, -0.014961985, 0.012728676, -0.021531789, -0.05905329, -0.06944409, -0.016700277, -0.060913816, -0.019436076, 0.08982007, 0.10389332, -0.102554955, 0.06951776, 0.07058402, -0.008742106, -0.02447418, 0.018724779, 0.028184239, 0.026121942, 0.013996168, 0.0664105, -0.038234126, -0.02303688, 0.04389385, 0.05041244, -0.11171202, 0.086676404, 0.028816389, -0.029978214, 0.043524325, -0.065447174, -0.07619985, -0.018115913, -0.066899836, -0.040868245, -0.10142577, 0.052631214, 0.039323542, -0.09457685, 0.027902136, -0.003998986, -0.037779003, 0.051883016, 3.1396424E-4, -0.03186952, -0.029589364, -0.024453204, 0.009792744, 0.002296945, -0.04543461, -0.034005456, 0.043245226, 0.038796358, -0.07531766, -0.034181666, -0.069799244, 0.001988578, 0.051464427, -0.09935535, -0.030321337, 0.010854493, 0.052270748, 0.041293442, -0.11673548, -0.034175783, 0.061309226, -0.06036236, 0.07255635, 0.009921444, -0.035960246, -0.047523238, -0.009834568, -0.040689506, -0.018173918, 0.011758854, 0.06473035, -0.023727052, 0.01765984, -0.0059596286, 0.023125466, -0.03085419, -0.018143924, 0.09201374, 0.009294985, -0.03208583, 0.012164934, 0.07489599, -0.06952085, -0.055458255, -0.04811663, 0.02467739, 0.014218112, 0.010280043, 4.3671473E-4, -0.0394551, 0.041456345, 0.0013710422, -0.009683716, 0.018802414, -0.006729811, -0.009802281, 0.018261077, 0.052742865, -0.022915887, 0.06353675, -0.007284017, -0.002082828, 0.007783691, -0.03130499, 0.023800777, -0.02610662, -0.023864703, 0.0068920017, 0.019401008, -0.0056829136, -5.0605834E-4, -0.053986073, -0.065493464, -0.034092687, 0.04220041, 0.009476794, 0.050892465, 0.005893973, -0.027504334, 0.0028201283, 0.0027392025, 0.027911827, 0.018575583, -0.005915392, 0.026672399, -0.009604471, -0.02643193, 0.0013825679, 0.061390538, 0.08235732, 0.031154502, 0.037601363, -0.016688455, 0.038007755, 0.07273599, 0.014851937, -0.0076043806, -0.07303425, -0.03740918, 0.019913351, 2.030503E-4, 0.0068934127, -0.0034623286, -0.024364877, 0.020607062, -0.035610355, 0.003314935, -0.044780705, -0.05102975, 0.026802188, 0.013005619, -0.012983566, -0.05817688, 0.012086601, -0.07851603, -0.032840643, 0.015080899, 0.0048408033, -0.022132527, -0.043668665, -0.024322625, 0.024781957, -0.026795862, 0.017768756, 0.0576815, -0.031258337, -0.04034984, -0.0022927672, 0.06328568, -0.05691621, -0.029289957, 0.007887512, -0.025037454, 0.052291643, -0.0042550424, -0.002641092, -0.03615621, -0.00850641, 0.00269375, 0.009949909, -0.011824053, 0.003560511, 0.055153664, 0.026184153, -0.06295364, -0.0066079907, -0.014661856, 0.035361435, 0.04479921, -0.01436901, -0.025124198, -0.023145422, -0.022270745, 0.013288247, -0.02767609, -0.01146416, 0.0145266075, -0.0054158275, -0.010314086, 0.009966933, 0.01918325, -0.02135528, 0.022194697, 0.014061586, 0.021323303, -0.0012787688, -0.011617605, -0.044630684, -0.0014462662, -0.055514146, 0.013914377, -0.012164333, 0.012878909, 0.009302253, -0.037286866, 0.025333503, -0.04553513, -0.016631879, -0.027500989, -0.047244523, 0.030365756, -0.012928509, -0.056037493, 0.002111567, -3.445983E-4, 0.01656157, 0.023865225, 0.014607268, -0.03616821, 0.02551308, -0.009524459, -0.031222345, -0.02033182, 0.057622015, 0.019413684, -0.015354169, 2.039349E-4, 0.0052822824, -0.02027396, 0.0038262396, -0.032031342, -0.005577524, -0.006228494, 0.013997069, -0.026569499, -0.007623818, -0.026430082, -0.037221637, -0.0011455176, 0.026016934, 0.0051541305, -0.04175314, -0.024648841, -0.03386957, 0.024139646, -0.027392618, -0.028452821, -0.0047744913, 0.026683511, -0.043278668, -0.008025299, 0.0062434734, 0.023799315, 0.018116387, -0.017212955, 0.00675846, 0.017144578, -0.011222329, -0.047040876, 0.011713254, 0.030690365, 0.020916743, 0.024264934, 0.0026452346, -0.031266063, 0.018595783, 0.008722972, 0.0011967237, 0.0240953, 0.007855105, -0.027829738, -0.0018804515, 0.0045171697, 0.0012034165, -0.025808694, -0.031137438, 0.012954249, -0.02222534, 0.007469989, -0.03299767, 0.019790541, 0.019729232, 0.025672013, 0.00655929, 0.019496253, 0.005682889, 0.026479306, 0.0050754775, -0.0039409203, 0.06357067, -0.028487764, -0.03659115, -0.02870191, 0.00958703, -0.0073981844, 0.0051872428, 0.0025473204, 0.006272702, -0.002030124, -0.028699424, -0.010338337, -0.023391975, -0.010535413, 0.012972192, 0.030313427, -0.017857233, -0.02584283, -0.026900806, -0.0029144983, -0.005277483, 0.014036093, -0.019638328, 0.029866232, -0.013569687, -0.025121475, -0.027261615, 0.009427129, 0.06471093, 0.015230324, -0.0056302394, -0.011471548, 0.017689008, 0.004637535, 0.0011149658, -0.031982716, 0.022016572, -0.0110083725, -0.0042767446, -6.200085E-4, 0.056941316, 0.01131996, -0.0036733507, -0.009270783, -0.010277945, -0.03191185, -0.0146463355, -0.005157946, -0.029410489, 0.03908128, 0.036739152, -0.017181905, -0.017025935, -0.009147714, 0.028543074, -0.027474517, 0.012574309, -0.005393016, 0.017847693, 0.01726672, 0.010323364, -0.0135736335, 0.0069461744, -0.022242447, 0.0058118105, 0.020939272, 0.0045747184, 0.014999139, 0.003252502, -0.016419375, 0.031438574, 0.04617525, 0.00784853, -0.0012723412, -0.025017604, 0.029681016, -0.024697304, -0.0052957274, 0.009767601, 0.0051521296, -0.0012994702, -0.0017888923, 0.044997703, -0.02585113, 0.020166703, -0.026824266, 0.025578074, 0.021925531, 0.014210808, 0.039668396, -0.005146907, 0.002887164, -0.003076182, -0.006285401, -0.023103757, -0.030286463, -0.027537484, 0.0030390045, -0.013392114, -0.0031120274, -0.018732185, 2.4149122E-4, 0.025475236, -0.029740073, -0.0056005623, -0.026741868, 0.0055567245, -0.008290855, 0.006020938, 0.017970854, 0.003936837, 0.011447729, 0.011691844, -0.0012378974, -0.0013880703, 0.036667068, 0.0128294565, -0.001602585, -0.0053273225, -0.013628352, -0.020104848, -0.0025651231, 0.0027692064, 0.041127414, -0.03252928, 0.015430981, 0.014353925, -0.035179876, 0.004730329, 0.0032820883, 0.019320745, 0.0070202886, -0.040679436, -0.034837708, -0.008630759, 0.009457761, 0.004813033, -0.020194704, -0.0034597763, -0.006086866, -0.012982794, -0.012567925, 0.013816319, -0.018442733, -0.018542282, -0.017392524, 0.007060914, 0.005591555, 0.0038210428, 0.020253645, -0.022122532, -0.010383561, 0.0068959575, -0.020679463, 0.028952265, -0.0073548774, 0.011456289, 1.1035489E-4, 6.2163285E-4, -0.014207815, 0.011708225, 0.009786873, -0.020304127, 0.019790946, -0.004707331, -0.0043455944, -0.001971539, 0.021671792, 3.45811E-4, -0.009738243, -0.014583063, -0.023843648, -0.017617645, -0.0073720664, -0.029533107, -0.0017647085, 0.02123342, 0.015121319, -0.024342237, 0.0062186574, 0.009744628, 0.032493576, 0.019315507, 0.014841595, 0.006602987, -6.539175E-4, -0.012376351, 0.026424857, -0.011808081, 0.024625171, 0.054046284, 0.004188485, -0.0016174468, 0.017886845, -0.012949441, 0.027347645, 0.021280972, -8.869145E-4, -0.019538047, -0.01612249, -0.015336986, -0.01007112, -0.001580681, 1.2961272E-4, -0.009443218, -0.013082444, 0.0037698112, -0.0049349843, -0.015419478, -0.028887676, -0.015861291, -0.02972866, -0.025518926, 0.0011080995, 0.01872132, -0.007845809, -0.01142388, 0.0010394778, 0.0076191085, -0.03206946, 0.0028279186, 0.0252884, -0.015494946, -0.00487451, -0.03042514, 0.0056792204, 0.00557942, 0.01477863, -0.016854461, -0.016603416, 0.033179272, 0.018492555, -0.018943263, -0.019813849, -0.02169808, 0.026190136, -0.0026397225, 0.012093653, -0.020144835, -0.003413523, -0.019216709, -0.016063567, 0.01053034, -0.019057969, 0.009921677, 0.017265338, -0.008042013, -0.032106385, -0.0076573566, -0.011284133, 0.0016440647, -0.0074232114, -0.0061121606, -0.0039731245, 0.017772922, -0.006773471, -0.003257682, -0.014163464, -0.0053975377, -0.03454502, -0.005380314, 0.025113512, -0.037035916, 0.005860556, 0.009815752, 0.0013864046, -0.021046035, -0.010914981, 0.02522609, 0.003725368, 0.024608297, -0.008895582, 0.020630056, -0.02082611, 0.005259784, -0.004662828, 0.018079981, -0.018179085, -0.012555717, -0.03287497, 0.024182148, 9.2749647E-4, -0.002516815, 0.0031977864, 0.0034360823, 0.022472538, -0.0093675945, 0.006006379, 0.007345346, 0.005745967, 0.008020854, 0.018388424, 4.7079913E-4, 0.0043508816, 0.0032794503, -0.008247611, 6.538056E-4, 0.006181467, 0.026531989, -0.009177468, -0.01658023, -0.0015889334, 0.022452582, 0.035810877, 0.013010471, 0.0015583939, -0.018390406, 0.0031738477, -0.021265933, -0.008688021, -0.009019973, 0.0069699707, 0.005563408, 0.009978513, -0.005430095, -0.009088654, -0.00735871, -0.0022795217, 0.01065763, -0.019584652, 0.0039610327, -0.01614338, -0.014655342, -0.009314964, -0.009561065, -0.022128796, 0.0051152497, 7.2398904E-4, 0.011731552, 0.019381693, -0.0018754478, 0.015200723, 0.01111195, -0.0057720356, 0.011230107, -0.0033303176, -0.0077977283, -0.012010337, 0.009293151, 2.3450867E-4, 0.018484943, 0.0050424044, -0.002583624, 0.003367546, 0.0018279159, -0.0017556995, 0.010968927, 0.009355919, 0.024280738, -0.0028832199, 0.019350704, 0.0015977627, 0.003447935, 0.017883845, 0.014826211, -0.02617111, -0.024304014, -0.0041993647, -0.020207208, -0.008573829, 0.0033712676, -0.006179999, 0.007474309, 0.0029519126, -0.0014506467, -0.013240714, -0.016551293, 0.012099103, -0.0050255316, -0.0057253223, -0.0021310174, -0.008341446, 0.0041917237, -0.008869906, 0.011595749, -0.0016620221, 0.007877958, 0.008671747, 0.016892435, 0.008028487, -0.02293363, 0.0087649925, -0.03605261, -0.013018509, -0.011498792, 0.010009394, 0.010596368, -0.0097576715, 0.007993852, 0.010651219, 0.0043800627, -9.224701E-4, 0.040351115, -0.0011116522, -0.02665772, 0.013831533, 0.0020280543, 0.0116251325, -0.032511443, -0.0046799295, 0.008501904, 0.0048212362, 0.0026921472, -0.0062492155]}}}",
                                "time_in_nanos": 210936868,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1502322,
                                    "match": 0,
                                    "score_count": 62720,
                                    "next_doc_count": 62738,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "count_weight_count": 0,
                                    "score": 209333946,
                                    "build_scorer_count": 36,
                                    "create_weight": 4420,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 96180
                                }
                            }
                        ],
                        "rewrite_time": 5810,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 212661259,
                                "children": [
                                    {
                                        "name": "SimpleTopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 209636292
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 830357,
                    "breakdown": {
                        "load_stored_fields": 528004,
                        "load_source": 4790,
                        "load_stored_fields_count": 4,
                        "next_reader_count": 2,
                        "load_source_count": 4,
                        "next_reader": 79301
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 57580,
                            "breakdown": {
                                "process_count": 4,
                                "process": 26880,
                                "next_reader": 30700,
                                "next_reader_count": 2
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 15621,
                            "breakdown": {
                                "process_count": 4,
                                "process": 15421,
                                "next_reader": 200,
                                "next_reader_count": 2
                            },
                            "debug": {
                                "fast_path": 4
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1830,
                            "breakdown": {
                                "process_count": 4,
                                "process": 1430,
                                "next_reader": 400,
                                "next_reader_count": 2
                            }
                        }
                    ]
                }
            },
            {
                "id": "[9TeIad8DS7qgMWTXJgxOow][vectest][1]",
                "node_id": "9TeIad8DS7qgMWTXJgxOow",
                "shard_id": 1,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 206694583,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1485974,
                                    "match": 0,
                                    "score_count": 62345,
                                    "next_doc_count": 62363,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "count_weight_count": 0,
                                    "score": 205100988,
                                    "build_scorer_count": 36,
                                    "create_weight": 4350,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 103271
                                }
                            }
                        ],
                        "rewrite_time": 6010,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 209886163,
                                "children": [
                                    {
                                        "name": "SimpleTopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 206875441
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 691175,
                    "breakdown": {
                        "load_stored_fields": 441323,
                        "load_source": 3000,
                        "load_stored_fields_count": 3,
                        "next_reader_count": 3,
                        "load_source_count": 3,
                        "next_reader": 94841
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 52010,
                            "breakdown": {
                                "process_count": 3,
                                "process": 16170,
                                "next_reader": 35840,
                                "next_reader_count": 3
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 10130,
                            "breakdown": {
                                "process_count": 3,
                                "process": 9820,
                                "next_reader": 310,
                                "next_reader_count": 3
                            },
                            "debug": {
                                "fast_path": 3
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1530,
                            "breakdown": {
                                "process_count": 3,
                                "process": 1050,
                                "next_reader": 480,
                                "next_reader_count": 3
                            }
                        }
                    ]
                }
            },
            {
                "id": "[9TeIad8DS7qgMWTXJgxOow][vectest][2]",
                "node_id": "9TeIad8DS7qgMWTXJgxOow",
                "shard_id": 2,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 204587242,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1597930,
                                    "match": 0,
                                    "score_count": 62480,
                                    "next_doc_count": 62498,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "count_weight_count": 0,
                                    "score": 202895452,
                                    "build_scorer_count": 36,
                                    "create_weight": 4430,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 89430
                                }
                            }
                        ],
                        "rewrite_time": 6080,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 205361237,
                                "children": [
                                    {
                                        "name": "SimpleTopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 202348159
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": []
            },
            {
                "id": "[9TeIad8DS7qgMWTXJgxOow][vectest][3]",
                "node_id": "9TeIad8DS7qgMWTXJgxOow",
                "shard_id": 3,
                "index": "vectest",
                "cluster": "(local)",
                "searches": [
                    {
                        "query": [
                            {
                                "type": "ScriptScoreQuery",
                                "description": "",
                                "time_in_nanos": 204081469,
                                "breakdown": {
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "next_doc": 1493449,
                                    "match": 0,
                                    "score_count": 62448,
                                    "next_doc_count": 62466,
                                    "compute_max_score_count": 0,
                                    "compute_max_score": 0,
                                    "advance": 0,
                                    "advance_count": 0,
                                    "count_weight_count": 0,
                                    "score": 202513639,
                                    "build_scorer_count": 36,
                                    "create_weight": 2310,
                                    "shallow_advance": 0,
                                    "count_weight": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 72071
                                }
                            }
                        ],
                        "rewrite_time": 3210,
                        "collector": [
                            {
                                "name": "QueryPhaseCollector",
                                "reason": "search_query_phase",
                                "time_in_nanos": 207237001,
                                "children": [
                                    {
                                        "name": "SimpleTopScoreDocCollector",
                                        "reason": "search_top_hits",
                                        "time_in_nanos": 204158303
                                    }
                                ]
                            }
                        ]
                    }
                ],
                "aggregations": [],
                "fetch": {
                    "type": "fetch",
                    "description": "",
                    "time_in_nanos": 685536,
                    "breakdown": {
                        "load_stored_fields": 372722,
                        "load_source": 4150,
                        "load_stored_fields_count": 3,
                        "next_reader_count": 3,
                        "load_source_count": 3,
                        "next_reader": 94702
                    },
                    "debug": {
                        "stored_fields": [
                            "_id",
                            "_routing",
                            "_source"
                        ]
                    },
                    "children": [
                        {
                            "type": "FetchFieldsPhase",
                            "description": "",
                            "time_in_nanos": 58190,
                            "breakdown": {
                                "process_count": 3,
                                "process": 22310,
                                "next_reader": 35880,
                                "next_reader_count": 3
                            }
                        },
                        {
                            "type": "FetchSourcePhase",
                            "description": "",
                            "time_in_nanos": 13490,
                            "breakdown": {
                                "process_count": 3,
                                "process": 13190,
                                "next_reader": 300,
                                "next_reader_count": 3
                            },
                            "debug": {
                                "fast_path": 3
                            }
                        },
                        {
                            "type": "StoredFieldsPhase",
                            "description": "",
                            "time_in_nanos": 1660,
                            "breakdown": {
                                "process_count": 3,
                                "process": 1140,
                                "next_reader": 520,
                                "next_reader_count": 3
                            }
                        }
                    ]
                }
            }
        ]
    }

8.19.3 JVM Info

{
                "pid": 2550,
                "version": "24.0.2",
                "vm_name": "OpenJDK 64-Bit Server VM",
                "vm_version": "24.0.2+12-54",
                "vm_vendor": "Oracle Corporation",
                "using_bundled_jdk": true,
                "start_time_in_millis": 1757634388684,
                "mem": {
                    "heap_init_in_bytes": 4294967296,
                    "heap_max_in_bytes": 4294967296,
                    "non_heap_init_in_bytes": 7667712,
                    "non_heap_max_in_bytes": 0,
                    "direct_max_in_bytes": 2147483648
                },
                "gc_collectors": [
                    "G1 Young Generation",
                    "G1 Concurrent GC",
                    "G1 Old Generation"
                ],
                "memory_pools": [
                    "CodeHeap 'non-nmethods'",
                    "Metaspace",
                    "CodeHeap 'profiled nmethods'",
                    "Compressed Class Space",
                    "G1 Eden Space",
                    "G1 Old Gen",
                    "G1 Survivor Space",
                    "CodeHeap 'non-profiled nmethods'"
                ],
                "using_compressed_ordinary_object_pointers": "true",
                "input_arguments": [
                    "-Des.networkaddress.cache.ttl=60",
                    "-Des.networkaddress.cache.negative.ttl=10",
                    "-XX:+AlwaysPreTouch",
                    "-Xss1m",
                    "-Djava.awt.headless=true",
                    "-Dfile.encoding=UTF-8",
                    "-Djna.nosys=true",
                    "-XX:-OmitStackTraceInFastThrow",
                    "-Dio.netty.noUnsafe=true",
                    "-Dio.netty.noKeySetOptimization=true",
                    "-Dio.netty.recycler.maxCapacityPerThread=0",
                    "-Dlog4j.shutdownHookEnabled=false",
                    "-Dlog4j2.disable.jmx=true",
                    "-Dlog4j2.formatMsgNoLookups=true",
                    "-Djava.locale.providers=CLDR",
                    "-Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24",
                    "-Des.path.home=/usr/share/elasticsearch",
                    "-Des.distribution.type=deb",
                    "-Des.java.type=bundled JDK",
                    "--enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core",
                    "--enable-native-access=ALL-UNNAMED",
                    "--illegal-native-access=deny",
                    "-XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log",
                    "-Des.entitlements.enabled=true",
                    "-XX:+EnableDynamicAgentLoading",
                    "-Djdk.attach.allowAttachSelf=true",
                    "--patch-module=java.base=/usr/share/elasticsearch/lib/entitlement-bridge/elasticsearch-entitlement-bridge-8.19.3.jar",
                    "--add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net",
                    "-Xms4g",
                    "-Xmx4g",
                    "-XX:+UseG1GC",
                    "-Djava.io.tmpdir=/tmp/elasticsearch-11894373277608162061",
                    "--add-modules=jdk.incubator.vector",
                    "-XX:+HeapDumpOnOutOfMemoryError",
                    "-XX:+ExitOnOutOfMemoryError",
                    "-XX:ErrorFile=hs_err_pid%p.log",
                    "-Xlog:gc*,gc+age=trace,safepoint:file=gc.log:utctime,level,pid,tags:filecount=8,filesize=64m",
                    "-XX:+UseCompressedOops",
                    "-Des.search.ignore_awareness_attributes=true",
                    "-Dlog4j2.formatMsgNoLookups=true",
                    "-XX:MaxDirectMemorySize=2147483648",
                    "-XX:G1HeapRegionSize=4m",
                    "-XX:InitiatingHeapOccupancyPercent=30",
                    "-XX:G1ReservePercent=15",
                    "--module-path=/usr/share/elasticsearch/lib",
                    "--add-modules=jdk.net",
                    "--add-modules=jdk.management.agent",
                    "--add-modules=ALL-MODULE-PATH",
                    "-Djdk.module.main=org.elasticsearch.server"
                ]
            }

Thanks for sharing @glund0. Looking at the query profile and JVM options you've sent, I see 2 key differences:

  1. The JVM options include the dense vector off-heap stats changes. I don't think this is the issue.
  2. I see that a new scorer is being used in the query phase. I believe it relates to this change where the score mode is inferred.

Having a dig through the release notes and docs I see that the as of 9.1.0, the index type for float vectors with greater than or equal to 384 dimensions is bbq_hnsw (see here).

Do you want to try updating the index_options type as see if that helps?

For both 8.19.3 and 9.1.3, I had index set to false, so I tried with index set to true and tried few different index_options.type settings. The type doesn’t seem to matter much for the performance:

8.19.3: 526 ms
9.1.3: 527 ms

But the behavior with index false is quite different:
8.19.3: 275 ms
9.1.3: 670 ms

The 8.19.3 behavior makes sense to me: turning on indexing makes the vectors larger to enable the kNN search. That makes it slower in the case where kNN isn’t being used. 9.1.3 surprises me because it’s much slower with indexing turned off.

On the JVM options, I suspect you’re correct that those don’t have any real impact on performance. Looks like those options are in code in the server-cli and I didn’t have any luck trying to start the server directly, so I couldn’t test that.

Any other options I could try? We’d like to upgrade to the latest version rather than stop at 8.19.3.

Hey @glund0

We tried to reproduce, but didn’t see the same regression. We are trying to do your “two vector” script now along with your specific settings.

The type doesn’t seem to matter much for the performance:

8.19.3: 526 ms
9.1.3: 527 ms

But the behavior with index false is quite different:
8.19.3: 275 ms
9.1.3: 670 ms

Awesome data point! Thanks! We will add this to our replication.

I have some questions in the mean time.

  • Are these two clusters that are just indexed independently? Meaning the data was indexed in version 9.1.3 and searched there, or was the cluster updated?
  • Could you return the segment count for both indices? (GET /{index}/_segments). Basically, we just need to rule out any weirdness (I doubt segment count is the cause).

This is running on a cluster with a single AWS r7a.xlarge data node, 4 GB JVM heap (~28 GB available for disk cache)

I will see if we can replicate on a similar node. I think we missed this part when trying to replicate internally.

I have two clusters running: one using 8.19.3 and one with 9.1.3. Both were created for performance testing, so they are new with no existing data. Both were loaded with the same data: a random sample of 250k records from our production environment

9.1.3 _segments

{
        "vectest": {
            "shards": {
                "0": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "C6ztsNfJS56-Nbzhk-Co1Q"
                        },
                        "num_committed_segments": 19,
                        "num_search_segments": 19,
                        "segments": {
                            "_1": {
                                "generation": 1,
                                "num_docs": 505,
                                "deleted_docs": 0,
                                "size_in_bytes": 9857550,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_a": {
                                "generation": 10,
                                "num_docs": 463,
                                "deleted_docs": 0,
                                "size_in_bytes": 9038818,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_c": {
                                "generation": 12,
                                "num_docs": 2697,
                                "deleted_docs": 0,
                                "size_in_bytes": 52637111,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 294,
                                "deleted_docs": 0,
                                "size_in_bytes": 5741150,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 280,
                                "deleted_docs": 0,
                                "size_in_bytes": 5466181,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 460,
                                "deleted_docs": 0,
                                "size_in_bytes": 8976063,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 249,
                                "deleted_docs": 0,
                                "size_in_bytes": 4861940,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 244,
                                "deleted_docs": 0,
                                "size_in_bytes": 4765162,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 259,
                                "deleted_docs": 0,
                                "size_in_bytes": 5057353,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 263,
                                "deleted_docs": 0,
                                "size_in_bytes": 5136241,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 250,
                                "deleted_docs": 0,
                                "size_in_bytes": 4881583,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 517,
                                "deleted_docs": 0,
                                "size_in_bytes": 10091222,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 9418,
                                "deleted_docs": 0,
                                "size_in_bytes": 183775630,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 569,
                                "deleted_docs": 0,
                                "size_in_bytes": 11104537,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 16589,
                                "deleted_docs": 0,
                                "size_in_bytes": 323692835,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 273,
                                "deleted_docs": 0,
                                "size_in_bytes": 5329236,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_q": {
                                "generation": 26,
                                "num_docs": 16522,
                                "deleted_docs": 0,
                                "size_in_bytes": 322388380,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_r": {
                                "generation": 27,
                                "num_docs": 442,
                                "deleted_docs": 0,
                                "size_in_bytes": 8629583,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_s": {
                                "generation": 28,
                                "num_docs": 12426,
                                "deleted_docs": 0,
                                "size_in_bytes": 242471767,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "1": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "C6ztsNfJS56-Nbzhk-Co1Q"
                        },
                        "num_committed_segments": 19,
                        "num_search_segments": 19,
                        "segments": {
                            "_1": {
                                "generation": 1,
                                "num_docs": 453,
                                "deleted_docs": 0,
                                "size_in_bytes": 8843237,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_6": {
                                "generation": 6,
                                "num_docs": 539,
                                "deleted_docs": 0,
                                "size_in_bytes": 10522743,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_8": {
                                "generation": 8,
                                "num_docs": 120,
                                "deleted_docs": 0,
                                "size_in_bytes": 2344874,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_a": {
                                "generation": 10,
                                "num_docs": 548,
                                "deleted_docs": 0,
                                "size_in_bytes": 10697634,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 2637,
                                "deleted_docs": 0,
                                "size_in_bytes": 51465896,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 410,
                                "deleted_docs": 0,
                                "size_in_bytes": 8003500,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 380,
                                "deleted_docs": 0,
                                "size_in_bytes": 7417626,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 258,
                                "deleted_docs": 0,
                                "size_in_bytes": 5036879,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 273,
                                "deleted_docs": 0,
                                "size_in_bytes": 5331001,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 267,
                                "deleted_docs": 0,
                                "size_in_bytes": 5215436,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 236,
                                "deleted_docs": 0,
                                "size_in_bytes": 4609048,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 477,
                                "deleted_docs": 0,
                                "size_in_bytes": 9313215,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 9284,
                                "deleted_docs": 0,
                                "size_in_bytes": 181154058,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 572,
                                "deleted_docs": 0,
                                "size_in_bytes": 11164773,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 17196,
                                "deleted_docs": 0,
                                "size_in_bytes": 335540690,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 543,
                                "deleted_docs": 0,
                                "size_in_bytes": 10598259,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_q": {
                                "generation": 26,
                                "num_docs": 16174,
                                "deleted_docs": 0,
                                "size_in_bytes": 315593192,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_r": {
                                "generation": 27,
                                "num_docs": 487,
                                "deleted_docs": 0,
                                "size_in_bytes": 9506165,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_s": {
                                "generation": 28,
                                "num_docs": 11491,
                                "deleted_docs": 0,
                                "size_in_bytes": 224225678,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "2": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "C6ztsNfJS56-Nbzhk-Co1Q"
                        },
                        "num_committed_segments": 19,
                        "num_search_segments": 19,
                        "segments": {
                            "_7": {
                                "generation": 7,
                                "num_docs": 224,
                                "deleted_docs": 0,
                                "size_in_bytes": 4373362,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_b": {
                                "generation": 11,
                                "num_docs": 259,
                                "deleted_docs": 0,
                                "size_in_bytes": 5055555,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_c": {
                                "generation": 12,
                                "num_docs": 3585,
                                "deleted_docs": 0,
                                "size_in_bytes": 69960782,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 246,
                                "deleted_docs": 0,
                                "size_in_bytes": 4804807,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 217,
                                "deleted_docs": 0,
                                "size_in_bytes": 4240152,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 400,
                                "deleted_docs": 0,
                                "size_in_bytes": 7806857,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 304,
                                "deleted_docs": 0,
                                "size_in_bytes": 5935966,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 261,
                                "deleted_docs": 0,
                                "size_in_bytes": 5096923,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 227,
                                "deleted_docs": 0,
                                "size_in_bytes": 4434313,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 206,
                                "deleted_docs": 0,
                                "size_in_bytes": 4023633,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 249,
                                "deleted_docs": 0,
                                "size_in_bytes": 4861952,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 465,
                                "deleted_docs": 0,
                                "size_in_bytes": 9076834,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 8756,
                                "deleted_docs": 0,
                                "size_in_bytes": 170854799,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 439,
                                "deleted_docs": 0,
                                "size_in_bytes": 8568778,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 17116,
                                "deleted_docs": 0,
                                "size_in_bytes": 333984921,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 546,
                                "deleted_docs": 0,
                                "size_in_bytes": 10659779,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_q": {
                                "generation": 26,
                                "num_docs": 15352,
                                "deleted_docs": 0,
                                "size_in_bytes": 299564200,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_r": {
                                "generation": 27,
                                "num_docs": 530,
                                "deleted_docs": 0,
                                "size_in_bytes": 10344706,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_s": {
                                "generation": 28,
                                "num_docs": 13098,
                                "deleted_docs": 0,
                                "size_in_bytes": 255562690,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "3": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "C6ztsNfJS56-Nbzhk-Co1Q"
                        },
                        "num_committed_segments": 19,
                        "num_search_segments": 19,
                        "segments": {
                            "_1": {
                                "generation": 1,
                                "num_docs": 519,
                                "deleted_docs": 0,
                                "size_in_bytes": 10128405,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_6": {
                                "generation": 6,
                                "num_docs": 458,
                                "deleted_docs": 0,
                                "size_in_bytes": 8941040,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_a": {
                                "generation": 10,
                                "num_docs": 476,
                                "deleted_docs": 0,
                                "size_in_bytes": 9294205,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 2266,
                                "deleted_docs": 0,
                                "size_in_bytes": 44229780,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 214,
                                "deleted_docs": 0,
                                "size_in_bytes": 4178846,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 370,
                                "deleted_docs": 0,
                                "size_in_bytes": 7223721,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 427,
                                "deleted_docs": 0,
                                "size_in_bytes": 8336800,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 237,
                                "deleted_docs": 0,
                                "size_in_bytes": 4629169,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 241,
                                "deleted_docs": 0,
                                "size_in_bytes": 4706118,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 264,
                                "deleted_docs": 0,
                                "size_in_bytes": 5154596,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 265,
                                "deleted_docs": 0,
                                "size_in_bytes": 5174718,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 541,
                                "deleted_docs": 0,
                                "size_in_bytes": 10560067,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 8401,
                                "deleted_docs": 0,
                                "size_in_bytes": 163930113,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 462,
                                "deleted_docs": 0,
                                "size_in_bytes": 9019193,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 13570,
                                "deleted_docs": 0,
                                "size_in_bytes": 264774287,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 209,
                                "deleted_docs": 0,
                                "size_in_bytes": 4082113,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_q": {
                                "generation": 26,
                                "num_docs": 15614,
                                "deleted_docs": 0,
                                "size_in_bytes": 304677974,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_r": {
                                "generation": 27,
                                "num_docs": 511,
                                "deleted_docs": 0,
                                "size_in_bytes": 9977209,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_s": {
                                "generation": 28,
                                "num_docs": 17403,
                                "deleted_docs": 0,
                                "size_in_bytes": 339580045,
                                "committed": true,
                                "search": true,
                                "version": "10.2.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ]
            }
        }
    }

8.19.3 _segments:

{
        "vectest": {
            "shards": {
                "0": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "0GfyfEfxRcmhBOtaaAnsNg"
                        },
                        "num_committed_segments": 16,
                        "num_search_segments": 16,
                        "segments": {
                            "_9": {
                                "generation": 9,
                                "num_docs": 257,
                                "deleted_docs": 0,
                                "size_in_bytes": 5424711,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_b": {
                                "generation": 11,
                                "num_docs": 235,
                                "deleted_docs": 0,
                                "size_in_bytes": 4958070,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_c": {
                                "generation": 12,
                                "num_docs": 3721,
                                "deleted_docs": 0,
                                "size_in_bytes": 78523001,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 219,
                                "deleted_docs": 0,
                                "size_in_bytes": 4622648,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 454,
                                "deleted_docs": 0,
                                "size_in_bytes": 9579290,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 248,
                                "deleted_docs": 0,
                                "size_in_bytes": 5233323,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 244,
                                "deleted_docs": 0,
                                "size_in_bytes": 5149157,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 8045,
                                "deleted_docs": 0,
                                "size_in_bytes": 169811834,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 370,
                                "deleted_docs": 0,
                                "size_in_bytes": 7804614,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 12572,
                                "deleted_docs": 0,
                                "size_in_bytes": 265388825,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 530,
                                "deleted_docs": 0,
                                "size_in_bytes": 11179579,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 13589,
                                "deleted_docs": 0,
                                "size_in_bytes": 286843794,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 243,
                                "deleted_docs": 0,
                                "size_in_bytes": 5128933,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 14157,
                                "deleted_docs": 0,
                                "size_in_bytes": 298873256,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 467,
                                "deleted_docs": 0,
                                "size_in_bytes": 9851024,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 7369,
                                "deleted_docs": 0,
                                "size_in_bytes": 155535817,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "1": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "0GfyfEfxRcmhBOtaaAnsNg"
                        },
                        "num_committed_segments": 16,
                        "num_search_segments": 16,
                        "segments": {
                            "_0": {
                                "generation": 0,
                                "num_docs": 183,
                                "deleted_docs": 0,
                                "size_in_bytes": 3863473,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_3": {
                                "generation": 3,
                                "num_docs": 502,
                                "deleted_docs": 0,
                                "size_in_bytes": 10589012,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_a": {
                                "generation": 10,
                                "num_docs": 465,
                                "deleted_docs": 0,
                                "size_in_bytes": 9812398,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 3001,
                                "deleted_docs": 0,
                                "size_in_bytes": 63324084,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 527,
                                "deleted_docs": 0,
                                "size_in_bytes": 11117425,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 263,
                                "deleted_docs": 0,
                                "size_in_bytes": 5550263,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 210,
                                "deleted_docs": 0,
                                "size_in_bytes": 4431058,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 6693,
                                "deleted_docs": 0,
                                "size_in_bytes": 141281173,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 720,
                                "deleted_docs": 0,
                                "size_in_bytes": 15185574,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 12593,
                                "deleted_docs": 0,
                                "size_in_bytes": 265835479,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 516,
                                "deleted_docs": 0,
                                "size_in_bytes": 10887698,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 13593,
                                "deleted_docs": 0,
                                "size_in_bytes": 286955872,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 262,
                                "deleted_docs": 0,
                                "size_in_bytes": 5528455,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 14259,
                                "deleted_docs": 0,
                                "size_in_bytes": 301007364,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 491,
                                "deleted_docs": 0,
                                "size_in_bytes": 10357990,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 8067,
                                "deleted_docs": 0,
                                "size_in_bytes": 170301676,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "2": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "0GfyfEfxRcmhBOtaaAnsNg"
                        },
                        "num_committed_segments": 16,
                        "num_search_segments": 16,
                        "segments": {
                            "_6": {
                                "generation": 6,
                                "num_docs": 180,
                                "deleted_docs": 0,
                                "size_in_bytes": 3800051,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_a": {
                                "generation": 10,
                                "num_docs": 395,
                                "deleted_docs": 0,
                                "size_in_bytes": 8331832,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_c": {
                                "generation": 12,
                                "num_docs": 2596,
                                "deleted_docs": 0,
                                "size_in_bytes": 54784427,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 224,
                                "deleted_docs": 0,
                                "size_in_bytes": 4727189,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 296,
                                "deleted_docs": 0,
                                "size_in_bytes": 6246606,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 415,
                                "deleted_docs": 0,
                                "size_in_bytes": 8753844,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 258,
                                "deleted_docs": 0,
                                "size_in_bytes": 5445658,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 7284,
                                "deleted_docs": 0,
                                "size_in_bytes": 153741300,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 535,
                                "deleted_docs": 0,
                                "size_in_bytes": 11285115,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 14069,
                                "deleted_docs": 0,
                                "size_in_bytes": 297005301,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 448,
                                "deleted_docs": 0,
                                "size_in_bytes": 9452173,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 14004,
                                "deleted_docs": 0,
                                "size_in_bytes": 295581140,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 761,
                                "deleted_docs": 0,
                                "size_in_bytes": 16049713,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 13712,
                                "deleted_docs": 0,
                                "size_in_bytes": 289399284,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 714,
                                "deleted_docs": 0,
                                "size_in_bytes": 15064069,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 6589,
                                "deleted_docs": 0,
                                "size_in_bytes": 139056625,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ],
                "3": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "0GfyfEfxRcmhBOtaaAnsNg"
                        },
                        "num_committed_segments": 16,
                        "num_search_segments": 16,
                        "segments": {
                            "_6": {
                                "generation": 6,
                                "num_docs": 244,
                                "deleted_docs": 0,
                                "size_in_bytes": 5149844,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_8": {
                                "generation": 8,
                                "num_docs": 155,
                                "deleted_docs": 0,
                                "size_in_bytes": 3272106,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_9": {
                                "generation": 9,
                                "num_docs": 247,
                                "deleted_docs": 0,
                                "size_in_bytes": 5213296,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_d": {
                                "generation": 13,
                                "num_docs": 3376,
                                "deleted_docs": 0,
                                "size_in_bytes": 71240361,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_e": {
                                "generation": 14,
                                "num_docs": 270,
                                "deleted_docs": 0,
                                "size_in_bytes": 5696707,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_f": {
                                "generation": 15,
                                "num_docs": 527,
                                "deleted_docs": 0,
                                "size_in_bytes": 11116687,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_g": {
                                "generation": 16,
                                "num_docs": 288,
                                "deleted_docs": 0,
                                "size_in_bytes": 6075278,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_h": {
                                "generation": 17,
                                "num_docs": 6367,
                                "deleted_docs": 0,
                                "size_in_bytes": 134382874,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_i": {
                                "generation": 18,
                                "num_docs": 510,
                                "deleted_docs": 0,
                                "size_in_bytes": 10760241,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_j": {
                                "generation": 19,
                                "num_docs": 14515,
                                "deleted_docs": 0,
                                "size_in_bytes": 306413401,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_k": {
                                "generation": 20,
                                "num_docs": 726,
                                "deleted_docs": 0,
                                "size_in_bytes": 15315282,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_l": {
                                "generation": 21,
                                "num_docs": 13765,
                                "deleted_docs": 0,
                                "size_in_bytes": 290556540,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_m": {
                                "generation": 22,
                                "num_docs": 502,
                                "deleted_docs": 0,
                                "size_in_bytes": 10590302,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_n": {
                                "generation": 23,
                                "num_docs": 14118,
                                "deleted_docs": 0,
                                "size_in_bytes": 297949867,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_o": {
                                "generation": 24,
                                "num_docs": 512,
                                "deleted_docs": 0,
                                "size_in_bytes": 10800113,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },
                            "_p": {
                                "generation": 25,
                                "num_docs": 6326,
                                "deleted_docs": 0,
                                "size_in_bytes": 133525824,
                                "committed": true,
                                "search": true,
                                "version": "9.12.2",
                                "compound": true,
                                "attributes": {
                                    "Lucene90StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            }
                        }
                    }
                ]
            }
        }
    }

Hey @glund0 ! Thanks for reporting this and all the info provided!

Segments do look pretty similar, indeed. We’re trying to reproduce your findings, using also a modified version of the so_vector rally track (rally-tracks/so_vector at master · elastic/rally-tracks · GitHub) but so far we’ve been unable to do so. We’re waiting on some results from an AWS node similar to your setup to see if this is reproducible in this case.

In the meantime, could you please:

  • Share the full query that you perform on the cluster (does it include any additional aggregations/field info etc)?

  • Check whether the issue is specific to your dataset, i.e. if you use a random generator or a different dataset with the same experiment params, is the difference in performance still there?

  • If possible, use JFR during the performance test to monitor what exactly goes on, and where most of the time is spent.

I’ll try using the so_vector project and generating random vectors with my setup to see if there’s any difference in performance. Below is the whole test query: there’s nothing besides the script_score.

The query is too big a forum post because of the vectors. So here’s the query with the first vector and I’ll post the second vector in a separate post

{
    "query": {
        "script_score": {
            "query": {
                "match_all": {}
            },
            "script": {
                "source": "Math.max(cosineSimilarity(params.vectorOne, 'vectorOne') + cosineSimilarity(params.vectorTwo, 'vectorTwo'), 0)",
                "params": {
                    "vectorOne": [
                        0.013173203,
                        0.0052297856,
                        0.21519354,
                        -0.093842305,
                        -0.01095355,
                        -0.29436445,
                        0.06493544,
                        0.05938409,
                        -0.17600165,
                        -0.20461452,
                        -0.015728287,
                        0.0046440605,
                        -0.030841334,
                        0.122089006,
                        -0.3034737,
                        0.037594393,
                        -0.10002559,
                        0.11257306,
                        -0.111819915,
                        -0.12071488,
                        0.020595897,
                        0.15776865,
                        0.052580394,
                        -0.09745094,
                        0.079588,
                        -0.16879044,
                        -0.034594744,
                        -0.050949633,
                        0.08187252,
                        -0.16822045,
                        -0.052350275,
                        0.048628043,
                        0.07935129,
                        0.03799321,
                        0.10957204,
                        0.028511189,
                        0.17220527,
                        -0.12096245,
                        -0.066655464,
                        0.08931657,
                        -0.054182395,
                        0.12907726,
                        -0.03926433,
                        0.058102854,
                        -0.17350669,
                        0.03771998,
                        -0.08872518,
                        0.011460013,
                        -0.01628334,
                        0.07906166,
                        -0.05854619,
                        -0.05143622,
                        -0.09604408,
                        -0.09826985,
                        0.045209,
                        0.05044051,
                        0.02117565,
                        -0.057305027,
                        0.04241245,
                        0.043347206,
                        0.07159769,
                        0.018550502,
                        0.04424336,
                        0.06912926,
                        -0.009729412,
                        0.03211654,
                        0.04782096,
                        -0.08818417,
                        -0.035109524,
                        0.018651119,
                        0.03159385,
                        0.022730356,
                        -0.020502992,
                        -0.066785574,
                        -0.029437877,
                        0.018001571,
                        -0.08194266,
                        -0.056691643,
                        0.027324012,
                        0.025953125,
                        0.058490414,
                        0.031390805,
                        -0.04929738,
                        0.02078965,
                        -0.07834581,
                        0.03741648,
                        -0.064613,
                        -0.0044922354,
                        0.05150282,
                        -0.01043456,
                        -0.038883932,
                        0.0013263221,
                        -0.047345605,
                        -0.03669518,
                        -0.073673025,
                        0.055956032,
                        -0.0031711988,
                        -0.0107596265,
                        -0.053912353,
                        -0.012321016,
                        0.0052027903,
                        -0.05931239,
                        -0.01181956,
                        0.03305743,
                        -0.040960968,
                        0.03575864,
                        -0.05874545,
                        0.038328175,
                        -0.028642122,
                        0.1001079,
                        -0.03132359,
                        -0.061485533,
                        0.012711688,
                        -0.04792668,
                        -0.042671476,
                        0.021871619,
                        -0.013868987,
                        -0.097119875,
                        0.00046119967,
                        0.0038548494,
                        -0.030143088,
                        0.04376323,
                        -0.04252116,
                        -0.003311639,
                        -0.017443746,
                        -0.012340171,
                        -0.032708775,
                        0.035120785,
                        -0.011805166,
                        -0.01372578,
                        -0.01727542,
                        -0.0023261674,
                        -0.040456187,
                        0.073783614,
                        -0.041797835,
                        -0.008352981,
                        -0.094251916,
                        0.04974681,
                        0.03162338,
                        0.027761696,
                        0.03158548,
                        -0.042769466,
                        0.002413028,
                        -0.0028053017,
                        0.027332032,
                        -0.013260667,
                        -0.00033364186,
                        0.022478772,
                        0.018060198,
                        0.010788596,
                        0.009725766,
                        0.01791517,
                        -0.017308813,
                        -0.06276063,
                        -0.022030298,
                        -0.06577266,
                        -0.006024345,
                        0.04061552,
                        0.06734963,
                        0.0169959,
                        0.03508686,
                        -0.004924476,
                        -0.007150875,
                        0.004072924,
                        -0.02452725,
                        0.0026079526,
                        0.040827397,
                        0.02485745,
                        0.012311198,
                        0.04595321,
                        0.023110976,
                        -0.008308107,
                        -0.01349351,
                        -0.0027289912,
                        -0.020215554,
                        0.029206075,
                        0.008211485,
                        -0.0020285,
                        -0.007962775,
                        -0.0011311583,
                        0.052420493,
                        0.029335976,
                        -0.034268316,
                        0.027089413,
                        -0.0141544575,
                        -0.037790395,
                        -0.032876235,
                        -0.018181248,
                        0.051618215,
                        -0.02391149,
                        0.040739205,
                        0.022332313,
                        -0.022192521,
                        -0.0038999524,
                        0.0002665285,
                        -0.026159858,
                        -0.025313236,
                        -0.011715555,
                        -0.013942653,
                        0.009715638,
                        -0.06985348,
                        -0.03986098,
                        0.0470792,
                        0.0131037645,
                        0.0569468,
                        -0.034224372,
                        0.019203423,
                        -0.0026869678,
                        -0.017593842,
                        0.023995357,
                        -0.009984715,
                        0.010128069,
                        0.03743259,
                        -0.019950591,
                        0.023682652,
                        -0.014933193,
                        0.0012517767,
                        -0.022865748,
                        -0.001210017,
                        0.0015929559,
                        -0.02579681,
                        -0.0017350283,
                        0.018419998,
                        -0.03629099,
                        0.0019098803,
                        0.020044416,
                        0.0005909709,
                        0.019216752,
                        -0.024207087,
                        -0.0041541145,
                        0.014369621,
                        0.0037044152,
                        -0.015547903,
                        0.04410028,
                        0.039407343,
                        0.0063609094,
                        0.018452296,
                        0.010365752,
                        0.0038212172,
                        -0.017657751,
                        -0.013848915,
                        -0.03659765,
                        0.01227015,
                        0.032493267,
                        0.034269303,
                        -0.020378161,
                        -0.028054774,
                        -0.0023942995,
                        -0.006439075,
                        -0.0050714565,
                        0.019598685,
                        -0.024586435,
                        -0.023228964,
                        -0.008255578,
                        -0.022321187,
                        -0.010975467,
                        -0.013256906,
                        -0.032204706,
                        0.052073203,
                        0.0054438584,
                        -0.0017145255,
                        0.02141263,
                        -0.013922718,
                        -0.0385722,
                        0.015181433,
                        0.022171939,
                        -0.0053502037,
                        -0.028801499,
                        -0.012135697,
                        -0.022278443,
                        -0.017694848,
                        -0.012493399,
                        0.0019240456,
                        -0.025850736,
                        -0.0060191327,
                        0.03563002,
                        0.0033583178,
                        0.015258327,
                        -0.0051530795,
                        0.002398866,
                        0.020764885,
                        0.020225616,
                        -0.0060330224,
                        0.029354535,
                        -0.009926263,
                        -0.01331316,
                        0.0076695136,
                        -0.009092792,
                        -0.016557172,
                        0.009162733,
                        -0.008613232,
                        0.02971107,
                        -0.017906457,
                        -0.00066019327,
                        -0.021018507,
                        0.0013300253,
                        0.001467901,
                        0.004009506,
                        0.010682631,
                        0.014998421,
                        0.0375694,
                        -0.013299593,
                        -0.027290402,
                        -0.0019929102,
                        -0.026408222,
                        0.0068023177,
                        -0.02077405,
                        -0.0073583247,
                        0.002652146,
                        0.045631643,
                        0.0046209916,
                        -0.030087095,
                        0.003158622,
                        -0.0012270092,
                        0.007513162,
                        0.015369132,
                        0.0061028395,
                        -0.0050769504,
                        0.014694878,
                        0.016838461,
                        -0.002496522,
                        -0.003751302,
                        0.026715659,
                        -0.01169101,
                        -0.00583926,
                        -0.0021141928,
                        -0.00827598,
                        0.003451413,
                        0.0031923712,
                        0.01344758,
                        0.00936066,
                        -0.0023082562,
                        -0.0030353786,
                        -0.0055100536,
                        0.03127455,
                        0.014757174,
                        0.024522578,
                        0.011656378,
                        -0.017068626,
                        0.014585248,
                        -0.021363627,
                        -0.008750968,
                        0.013437441,
                        0.008366836,
                        0.0120168,
                        -0.011006667,
                        -0.0028149113,
                        -0.0076802806,
                        0.021877414,
                        0.01408847,
                        0.035885923,
                        -0.0072402735,
                        -0.008659223,
                        0.009754819,
                        0.008619722,
                        -0.0031400977,
                        -0.026133742,
                        -0.004457135,
                        0.018737331,
                        0.004529896,
                        -0.010196676,
                        0.019900426,
                        0.015067387,
                        0.03402317,
                        0.01999167,
                        -0.005758707,
                        0.00979926,
                        -0.005611558,
                        -0.0127474945,
                        0.014552781,
                        -0.016785823,
                        0.009479103,
                        0.02297833,
                        -0.010731883,
                        -0.016717015,
                        -0.017484287,
                        0.020361323,
                        -0.013998314,
                        0.017850846,
                        0.014292788,
                        0.048527926,
                        0.010552317,
                        -0.018746307,
                        -0.017717497,
                        0.012798664,
                        0.0037520204,
                        0.0064832903,
                        0.024817422,
                        -0.026294582,
                        0.0051326375,
                        -0.0074774167,
                        -0.0074702553,
                        0.0017218001,
                        0.00080103375,
                        0.005199527,
                        0.017515521,
                        -0.0066512986,
                        -0.0013254563,
                        0.0025569454,
                        -0.010042196,
                        -0.019547828,
                        0.00667863,
                        0.004732076,
                        -0.005076369,
                        -0.0051990394,
                        0.005639958,
                        -0.008840068,
                        -0.00202057,
                        0.004196233,
                        0.012377474,
                        0.011439983,
                        -0.008849425,
                        0.01082493,
                        -0.007176978,
                        0.0027949852,
                        0.001586264,
                        -0.020166483,
                        -0.019787777,
                        -0.0276374,
                        0.016102798,
                        -0.018139778,
                        0.009249267,
                        0.013843168,
                        -0.004218314,
                        -0.014982909,
                        0.017630966,
                        0.0018553302,
                        -0.010762763,
                        0.010247762,
                        0.020129021,
                        -0.02226605,
                        0.017677778,
                        -0.0136600705,
                        -0.021495359,
                        0.0009576419,
                        -0.0026726376,
                        0.0026236682,
                        0.009962605,
                        -0.018459925,
                        -0.021139728,
                        0.007942952,
                        -0.01841508,
                        0.016816823,
                        -0.0120083615,
                        -0.020876758,
                        0.017487485,
                        -0.011791492,
                        0.023464072,
                        0.00016855022,
                        -0.019260358,
                        -0.015247996,
                        -0.020723242,
                        -0.0072092214,
                        0.0038836407,
                        -0.014971971,
                        -0.0067802165,
                        0.012383788,
                        0.003148533,
                        -0.001996739,
                        0.017085847,
                        0.01043297,
                        0.015849201,
                        0.022640757,
                        0.017766962,
                        -0.02684469,
                        0.02430105,
                        0.011285774,
                        0.0044941027,
                        0.016542362,
                        0.009292583,
                        0.01089254,
                        0.010973071,
                        0.009921301,
                        0.013253412,
                        0.018811962,
                        0.008307326,
                        0.004228023,
                        0.008613386,
                        -0.002340278,
                        0.019086504,
                        0.002332518,
                        -0.012299346,
                        3.020229e-05,
                        -0.007913956,
                        -0.003973171,
                        -0.017028747,
                        0.009567644,
                        0.008149756,
                        0.012768958,
                        0.001068468,
                        0.009620058,
                        -0.004871017,
                        -0.003550283,
                        0.012100322,
                        -0.015033302,
                        -0.008209837,
                        0.002292909,
                        0.023443846,
                        0.011615485,
                        0.0047445227,
                        -0.0009560393,
                        -0.000327645,
                        0.00909202,
                        0.021312146,
                        0.003922159,
                        -0.020681487,
                        -0.013827459,
                        -0.0049597328,
                        0.005719597,
                        0.003362033,
                        0.010312935,
                        0.0023698234,
                        -0.012395746,
                        -0.009986458,
                        -0.013381608,
                        0.009090864,
                        -0.010249262,
                        1.2023547e-05,
                        0.0038468342,
                        0.006337594,
                        0.0012656534,
                        -0.021973915,
                        0.013570165,
                        -0.002527013,
                        0.0025838292,
                        -0.01267369,
                        -0.01185739,
                        -0.0012393842,
                        -0.007108769,
                        0.019191096,
                        0.0076847733,
                        -8.00497e-05,
                        0.0062433304,
                        -0.0059921932,
                        -0.007990699,
                        0.0028100375,
                        -0.0067075496,
                        -0.016492415,
                        -0.01980759,
                        -0.015540144,
                        -0.0023266955,
                        0.0073061595,
                        -0.0009437309,
                        -0.0027223274,
                        -0.0017387761,
                        -0.01786714,
                        0.016558483,
                        0.004781822,
                        0.013613918,
                        0.005038345,
                        0.0025672123,
                        -0.004939192,
                        0.00478557,
                        -0.003682138,
                        -0.017883932,
                        -0.0085733775,
                        -0.0139945075,
                        -0.023469679,
                        -0.009169903,
                        -0.003148576,
                        -0.005561104,
                        -0.008181999,
                        0.00439613,
                        0.015162514,
                        0.0063574747,
                        0.016413234,
                        -0.004096887,
                        -0.012458653,
                        -0.018151276,
                        -0.004095136,
                        0.00926797,
                        -0.006555397,
                        0.008840552,
                        0.008436288,
                        -0.0044730115,
                        -0.017382707,
                        -0.0025450094,
                        -0.009555567,
                        0.0025377574,
                        0.011402533,
                        -0.01082082,
                        0.0048879785,
                        0.008872678,
                        0.0071776286,
                        -0.0028551456,
                        0.025974488,
                        -0.0031310457,
                        -0.021706022,
                        -0.009738637,
                        -0.000646065,
                        -0.00079623424,
                        -0.008404216,
                        0.003505369,
                        -0.0006689662,
                        -0.016984435,
                        -0.0036331743,
                        0.0063550733,
                        -0.004627766,
                        -0.0028516732,
                        0.006443027,
                        0.0020748319,
                        0.021670623,
                        -0.0057692537,
                        -0.003918615,
                        -0.0031876706,
                        -0.0027631929,
                        -0.009873856,
                        0.0024800557,
                        -0.008688225,
                        -0.0015063204,
                        -0.017516257,
                        -0.021662358,
                        0.0040566046,
                        0.001759627,
                        -0.00873719,
                        0.006285713,
                        0.014531575,
                        -0.011670351,
                        0.0009434051,
                        -0.00920805,
                        0.01510458,
                        -0.006492559,
                        0.011056852,
                        0.0018427926,
                        -3.124221e-05,
                        -0.015957613,
                        0.0021030544,
                        -0.008498496,
                        -0.017197903,
                        -0.01116062,
                        0.00774098,
                        0.0061622197,
                        -0.008144267,
                        0.009911781,
                        0.01610726,
                        -0.0066612447,
                        0.0033935746,
                        -0.014372329,
                        0.0045597753,
                        -0.020207006,
                        0.0066753887,
                        0.021799188,
                        0.0040517636,
                        0.0045109354,
                        0.011718117,
                        -0.00026437384,
                        -0.0005938372,
                        0.002140971,
                        -0.0017630563,
                        0.0048767035,
                        0.017311148,
                        -0.00504668,
                        -0.0147418175,
                        0.0139642665,
                        0.001257399,
                        0.008115203,
                        -0.0028234539,
                        0.005615775,
                        0.0020506762,
                        -0.00029385765,
                        -0.011488344,
                        -0.00048777653,
                        0.005667841,
                        0.009290322,
                        -0.012496263,
                        -0.0025094815,
                        -0.0024429646,
                        0.0090868,
                        0.003280879,
                        -0.0019263525,
                        0.0030596212,
                        0.00457934,
                        0.007970626,
                        0.009429205,
                        0.0039575854,
                        0.0011453362,
                        0.003737679,
                        0.0062191375,
                        -0.0033365435,
                        0.0055586784,
                        0.0046217036,
                        -0.0059943916,
                        0.0101486705,
                        0.012657837,
                        0.00048796192,
                        0.005385128,
                        0.0041495743,
                        -0.010394916,
                        -0.007758405,
                        -0.005766732,
                        0.011718328,
                        -0.01692151,
                        0.0009872961,
                        -0.002198138,
                        0.016635805,
                        0.0027692034,
                        -0.0050674435,
                        0.0011226385,
                        -0.0021900255,
                        -0.00097107905,
                        -0.005269831,
                        -0.007580122,
                        0.0032111898,
                        0.0020253612,
                        -0.0044059856,
                        -0.0037788139,
                        0.008817335,
                        -0.0062701693,
                        0.005308133,
                        -0.0013404146,
                        -0.0061498955,
                        0.0059099253,
                        -0.00045824097,
                        -0.0046645263,
                        -0.0020703068,
                        0.0010247212,
                        -0.00089980697,
                        0.014734301,
                        0.008145946,
                        0.001952657,
                        0.00049267797,
                        0.0004033092,
                        0.0016065567,
                        -0.005226729,
                        0.010558401,
                        -0.0063024457,
                        0.0018141957,
                        0.005426263,
                        0.009011342,
                        -0.0055915676,
                        0.0029521552,
                        -0.0030315302,
                        0.013015645,
                        -0.003965996,
                        -0.013479809,
                        0.011376977,
                        -0.005542539,
                        -0.012652742,
                        -0.001385992,
                        -0.018611869,
                        0.010869142,
                        0.0016872283,
                        0.0014726688,
                        0.008198611,
                        -0.0077167833,
                        -0.0104016615,
                        0.0032530972,
                        0.004940237,
                        -0.005863709,
                        0.012621446,
                        0.0061782463,
                        -0.012538319,
                        0.00094925676,
                        -0.008842992,
                        -0.006489274,
                        -0.0007138018,
                        0.007559196,
                        0.0016362873,
                        0.008001425,
                        0.0041235443,
                        -0.0012436371,
                        0.011159509,
                        0.0009762285,
                        0.009429023,
                        -0.00023953077,
                        0.009857336,
                        0.0067546843,
                        0.004515829
                    ],
                    "vectorTwo": [...]
                }
            }
        }
    }
}
"vectorTwo": [
                        -0.11973515,
                        0.08095747,
                        -7.9745345E-4,
                        0.017778162,
                        0.03781757,
                        0.0948174,
                        -0.010377076,
                        -0.03181369,
                        0.05628834,
                        0.05891907,
                        -0.030822765,
                        0.10757859,
                        -0.23740931,
                        -0.22470808,
                        0.1611373,
                        -0.13324434,
                        -0.10559932,
                        0.09629354,
                        -0.18414927,
                        -0.10610022,
                        0.108907506,
                        -0.014238788,
                        0.13708599,
                        -0.0932529,
                        -0.046637416,
                        -0.0723548,
                        0.021216057,
                        -0.048501376,
                        0.10892839,
                        -0.0034054436,
                        0.12404433,
                        0.11910919,
                        0.0407715,
                        0.030051203,
                        0.003348333,
                        0.048397772,
                        0.038554996,
                        -0.006637996,
                        0.02692062,
                        -0.05859029,
                        0.03757299,
                        0.06997503,
                        0.0697287,
                        -0.016306043,
                        0.050575297,
                        0.035764903,
                        -0.010194664,
                        -0.03432703,
                        -0.036454998,
                        -0.040881,
                        -0.0017276381,
                        0.119421884,
                        -0.077626854,
                        -0.01929169,
                        -0.019275935,
                        0.17408307,
                        0.04608272,
                        -0.02536122,
                        0.048946682,
                        0.026566505,
                        0.12054666,
                        -0.014961985,
                        0.012728676,
                        -0.021531789,
                        -0.05905329,
                        -0.06944409,
                        -0.016700277,
                        -0.060913816,
                        -0.019436076,
                        0.08982007,
                        0.10389332,
                        -0.102554955,
                        0.06951776,
                        0.07058402,
                        -0.008742106,
                        -0.02447418,
                        0.018724779,
                        0.028184239,
                        0.026121942,
                        0.013996168,
                        0.0664105,
                        -0.038234126,
                        -0.02303688,
                        0.04389385,
                        0.05041244,
                        -0.11171202,
                        0.086676404,
                        0.028816389,
                        -0.029978214,
                        0.043524325,
                        -0.065447174,
                        -0.07619985,
                        -0.018115913,
                        -0.066899836,
                        -0.040868245,
                        -0.10142577,
                        0.052631214,
                        0.039323542,
                        -0.09457685,
                        0.027902136,
                        -0.003998986,
                        -0.037779003,
                        0.051883016,
                        3.1396424E-4,
                        -0.03186952,
                        -0.029589364,
                        -0.024453204,
                        0.009792744,
                        0.002296945,
                        -0.04543461,
                        -0.034005456,
                        0.043245226,
                        0.038796358,
                        -0.07531766,
                        -0.034181666,
                        -0.069799244,
                        0.001988578,
                        0.051464427,
                        -0.09935535,
                        -0.030321337,
                        0.010854493,
                        0.052270748,
                        0.041293442,
                        -0.11673548,
                        -0.034175783,
                        0.061309226,
                        -0.06036236,
                        0.07255635,
                        0.009921444,
                        -0.035960246,
                        -0.047523238,
                        -0.009834568,
                        -0.040689506,
                        -0.018173918,
                        0.011758854,
                        0.06473035,
                        -0.023727052,
                        0.01765984,
                        -0.0059596286,
                        0.023125466,
                        -0.03085419,
                        -0.018143924,
                        0.09201374,
                        0.009294985,
                        -0.03208583,
                        0.012164934,
                        0.07489599,
                        -0.06952085,
                        -0.055458255,
                        -0.04811663,
                        0.02467739,
                        0.014218112,
                        0.010280043,
                        4.3671473E-4,
                        -0.0394551,
                        0.041456345,
                        0.0013710422,
                        -0.009683716,
                        0.018802414,
                        -0.006729811,
                        -0.009802281,
                        0.018261077,
                        0.052742865,
                        -0.022915887,
                        0.06353675,
                        -0.007284017,
                        -0.002082828,
                        0.007783691,
                        -0.03130499,
                        0.023800777,
                        -0.02610662,
                        -0.023864703,
                        0.0068920017,
                        0.019401008,
                        -0.0056829136,
                        -5.0605834E-4,
                        -0.053986073,
                        -0.065493464,
                        -0.034092687,
                        0.04220041,
                        0.009476794,
                        0.050892465,
                        0.005893973,
                        -0.027504334,
                        0.0028201283,
                        0.0027392025,
                        0.027911827,
                        0.018575583,
                        -0.005915392,
                        0.026672399,
                        -0.009604471,
                        -0.02643193,
                        0.0013825679,
                        0.061390538,
                        0.08235732,
                        0.031154502,
                        0.037601363,
                        -0.016688455,
                        0.038007755,
                        0.07273599,
                        0.014851937,
                        -0.0076043806,
                        -0.07303425,
                        -0.03740918,
                        0.019913351,
                        2.030503E-4,
                        0.0068934127,
                        -0.0034623286,
                        -0.024364877,
                        0.020607062,
                        -0.035610355,
                        0.003314935,
                        -0.044780705,
                        -0.05102975,
                        0.026802188,
                        0.013005619,
                        -0.012983566,
                        -0.05817688,
                        0.012086601,
                        -0.07851603,
                        -0.032840643,
                        0.015080899,
                        0.0048408033,
                        -0.022132527,
                        -0.043668665,
                        -0.024322625,
                        0.024781957,
                        -0.026795862,
                        0.017768756,
                        0.0576815,
                        -0.031258337,
                        -0.04034984,
                        -0.0022927672,
                        0.06328568,
                        -0.05691621,
                        -0.029289957,
                        0.007887512,
                        -0.025037454,
                        0.052291643,
                        -0.0042550424,
                        -0.002641092,
                        -0.03615621,
                        -0.00850641,
                        0.00269375,
                        0.009949909,
                        -0.011824053,
                        0.003560511,
                        0.055153664,
                        0.026184153,
                        -0.06295364,
                        -0.0066079907,
                        -0.014661856,
                        0.035361435,
                        0.04479921,
                        -0.01436901,
                        -0.025124198,
                        -0.023145422,
                        -0.022270745,
                        0.013288247,
                        -0.02767609,
                        -0.01146416,
                        0.0145266075,
                        -0.0054158275,
                        -0.010314086,
                        0.009966933,
                        0.01918325,
                        -0.02135528,
                        0.022194697,
                        0.014061586,
                        0.021323303,
                        -0.0012787688,
                        -0.011617605,
                        -0.044630684,
                        -0.0014462662,
                        -0.055514146,
                        0.013914377,
                        -0.012164333,
                        0.012878909,
                        0.009302253,
                        -0.037286866,
                        0.025333503,
                        -0.04553513,
                        -0.016631879,
                        -0.027500989,
                        -0.047244523,
                        0.030365756,
                        -0.012928509,
                        -0.056037493,
                        0.002111567,
                        -3.445983E-4,
                        0.01656157,
                        0.023865225,
                        0.014607268,
                        -0.03616821,
                        0.02551308,
                        -0.009524459,
                        -0.031222345,
                        -0.02033182,
                        0.057622015,
                        0.019413684,
                        -0.015354169,
                        2.039349E-4,
                        0.0052822824,
                        -0.02027396,
                        0.0038262396,
                        -0.032031342,
                        -0.005577524,
                        -0.006228494,
                        0.013997069,
                        -0.026569499,
                        -0.007623818,
                        -0.026430082,
                        -0.037221637,
                        -0.0011455176,
                        0.026016934,
                        0.0051541305,
                        -0.04175314,
                        -0.024648841,
                        -0.03386957,
                        0.024139646,
                        -0.027392618,
                        -0.028452821,
                        -0.0047744913,
                        0.026683511,
                        -0.043278668,
                        -0.008025299,
                        0.0062434734,
                        0.023799315,
                        0.018116387,
                        -0.017212955,
                        0.00675846,
                        0.017144578,
                        -0.011222329,
                        -0.047040876,
                        0.011713254,
                        0.030690365,
                        0.020916743,
                        0.024264934,
                        0.0026452346,
                        -0.031266063,
                        0.018595783,
                        0.008722972,
                        0.0011967237,
                        0.0240953,
                        0.007855105,
                        -0.027829738,
                        -0.0018804515,
                        0.0045171697,
                        0.0012034165,
                        -0.025808694,
                        -0.031137438,
                        0.012954249,
                        -0.02222534,
                        0.007469989,
                        -0.03299767,
                        0.019790541,
                        0.019729232,
                        0.025672013,
                        0.00655929,
                        0.019496253,
                        0.005682889,
                        0.026479306,
                        0.0050754775,
                        -0.0039409203,
                        0.06357067,
                        -0.028487764,
                        -0.03659115,
                        -0.02870191,
                        0.00958703,
                        -0.0073981844,
                        0.0051872428,
                        0.0025473204,
                        0.006272702,
                        -0.002030124,
                        -0.028699424,
                        -0.010338337,
                        -0.023391975,
                        -0.010535413,
                        0.012972192,
                        0.030313427,
                        -0.017857233,
                        -0.02584283,
                        -0.026900806,
                        -0.0029144983,
                        -0.005277483,
                        0.014036093,
                        -0.019638328,
                        0.029866232,
                        -0.013569687,
                        -0.025121475,
                        -0.027261615,
                        0.009427129,
                        0.06471093,
                        0.015230324,
                        -0.0056302394,
                        -0.011471548,
                        0.017689008,
                        0.004637535,
                        0.0011149658,
                        -0.031982716,
                        0.022016572,
                        -0.0110083725,
                        -0.0042767446,
                        -6.200085E-4,
                        0.056941316,
                        0.01131996,
                        -0.0036733507,
                        -0.009270783,
                        -0.010277945,
                        -0.03191185,
                        -0.0146463355,
                        -0.005157946,
                        -0.029410489,
                        0.03908128,
                        0.036739152,
                        -0.017181905,
                        -0.017025935,
                        -0.009147714,
                        0.028543074,
                        -0.027474517,
                        0.012574309,
                        -0.005393016,
                        0.017847693,
                        0.01726672,
                        0.010323364,
                        -0.0135736335,
                        0.0069461744,
                        -0.022242447,
                        0.0058118105,
                        0.020939272,
                        0.0045747184,
                        0.014999139,
                        0.003252502,
                        -0.016419375,
                        0.031438574,
                        0.04617525,
                        0.00784853,
                        -0.0012723412,
                        -0.025017604,
                        0.029681016,
                        -0.024697304,
                        -0.0052957274,
                        0.009767601,
                        0.0051521296,
                        -0.0012994702,
                        -0.0017888923,
                        0.044997703,
                        -0.02585113,
                        0.020166703,
                        -0.026824266,
                        0.025578074,
                        0.021925531,
                        0.014210808,
                        0.039668396,
                        -0.005146907,
                        0.002887164,
                        -0.003076182,
                        -0.006285401,
                        -0.023103757,
                        -0.030286463,
                        -0.027537484,
                        0.0030390045,
                        -0.013392114,
                        -0.0031120274,
                        -0.018732185,
                        2.4149122E-4,
                        0.025475236,
                        -0.029740073,
                        -0.0056005623,
                        -0.026741868,
                        0.0055567245,
                        -0.008290855,
                        0.006020938,
                        0.017970854,
                        0.003936837,
                        0.011447729,
                        0.011691844,
                        -0.0012378974,
                        -0.0013880703,
                        0.036667068,
                        0.0128294565,
                        -0.001602585,
                        -0.0053273225,
                        -0.013628352,
                        -0.020104848,
                        -0.0025651231,
                        0.0027692064,
                        0.041127414,
                        -0.03252928,
                        0.015430981,
                        0.014353925,
                        -0.035179876,
                        0.004730329,
                        0.0032820883,
                        0.019320745,
                        0.0070202886,
                        -0.040679436,
                        -0.034837708,
                        -0.008630759,
                        0.009457761,
                        0.004813033,
                        -0.020194704,
                        -0.0034597763,
                        -0.006086866,
                        -0.012982794,
                        -0.012567925,
                        0.013816319,
                        -0.018442733,
                        -0.018542282,
                        -0.017392524,
                        0.007060914,
                        0.005591555,
                        0.0038210428,
                        0.020253645,
                        -0.022122532,
                        -0.010383561,
                        0.0068959575,
                        -0.020679463,
                        0.028952265,
                        -0.0073548774,
                        0.011456289,
                        1.1035489E-4,
                        6.2163285E-4,
                        -0.014207815,
                        0.011708225,
                        0.009786873,
                        -0.020304127,
                        0.019790946,
                        -0.004707331,
                        -0.0043455944,
                        -0.001971539,
                        0.021671792,
                        3.45811E-4,
                        -0.009738243,
                        -0.014583063,
                        -0.023843648,
                        -0.017617645,
                        -0.0073720664,
                        -0.029533107,
                        -0.0017647085,
                        0.02123342,
                        0.015121319,
                        -0.024342237,
                        0.0062186574,
                        0.009744628,
                        0.032493576,
                        0.019315507,
                        0.014841595,
                        0.006602987,
                        -6.539175E-4,
                        -0.012376351,
                        0.026424857,
                        -0.011808081,
                        0.024625171,
                        0.054046284,
                        0.004188485,
                        -0.0016174468,
                        0.017886845,
                        -0.012949441,
                        0.027347645,
                        0.021280972,
                        -8.869145E-4,
                        -0.019538047,
                        -0.01612249,
                        -0.015336986,
                        -0.01007112,
                        -0.001580681,
                        1.2961272E-4,
                        -0.009443218,
                        -0.013082444,
                        0.0037698112,
                        -0.0049349843,
                        -0.015419478,
                        -0.028887676,
                        -0.015861291,
                        -0.02972866,
                        -0.025518926,
                        0.0011080995,
                        0.01872132,
                        -0.007845809,
                        -0.01142388,
                        0.0010394778,
                        0.0076191085,
                        -0.03206946,
                        0.0028279186,
                        0.0252884,
                        -0.015494946,
                        -0.00487451,
                        -0.03042514,
                        0.0056792204,
                        0.00557942,
                        0.01477863,
                        -0.016854461,
                        -0.016603416,
                        0.033179272,
                        0.018492555,
                        -0.018943263,
                        -0.019813849,
                        -0.02169808,
                        0.026190136,
                        -0.0026397225,
                        0.012093653,
                        -0.020144835,
                        -0.003413523,
                        -0.019216709,
                        -0.016063567,
                        0.01053034,
                        -0.019057969,
                        0.009921677,
                        0.017265338,
                        -0.008042013,
                        -0.032106385,
                        -0.0076573566,
                        -0.011284133,
                        0.0016440647,
                        -0.0074232114,
                        -0.0061121606,
                        -0.0039731245,
                        0.017772922,
                        -0.006773471,
                        -0.003257682,
                        -0.014163464,
                        -0.0053975377,
                        -0.03454502,
                        -0.005380314,
                        0.025113512,
                        -0.037035916,
                        0.005860556,
                        0.009815752,
                        0.0013864046,
                        -0.021046035,
                        -0.010914981,
                        0.02522609,
                        0.003725368,
                        0.024608297,
                        -0.008895582,
                        0.020630056,
                        -0.02082611,
                        0.005259784,
                        -0.004662828,
                        0.018079981,
                        -0.018179085,
                        -0.012555717,
                        -0.03287497,
                        0.024182148,
                        9.2749647E-4,
                        -0.002516815,
                        0.0031977864,
                        0.0034360823,
                        0.022472538,
                        -0.0093675945,
                        0.006006379,
                        0.007345346,
                        0.005745967,
                        0.008020854,
                        0.018388424,
                        4.7079913E-4,
                        0.0043508816,
                        0.0032794503,
                        -0.008247611,
                        6.538056E-4,
                        0.006181467,
                        0.026531989,
                        -0.009177468,
                        -0.01658023,
                        -0.0015889334,
                        0.022452582,
                        0.035810877,
                        0.013010471,
                        0.0015583939,
                        -0.018390406,
                        0.0031738477,
                        -0.021265933,
                        -0.008688021,
                        -0.009019973,
                        0.0069699707,
                        0.005563408,
                        0.009978513,
                        -0.005430095,
                        -0.009088654,
                        -0.00735871,
                        -0.0022795217,
                        0.01065763,
                        -0.019584652,
                        0.0039610327,
                        -0.01614338,
                        -0.014655342,
                        -0.009314964,
                        -0.009561065,
                        -0.022128796,
                        0.0051152497,
                        7.2398904E-4,
                        0.011731552,
                        0.019381693,
                        -0.0018754478,
                        0.015200723,
                        0.01111195,
                        -0.0057720356,
                        0.011230107,
                        -0.0033303176,
                        -0.0077977283,
                        -0.012010337,
                        0.009293151,
                        2.3450867E-4,
                        0.018484943,
                        0.0050424044,
                        -0.002583624,
                        0.003367546,
                        0.0018279159,
                        -0.0017556995,
                        0.010968927,
                        0.009355919,
                        0.024280738,
                        -0.0028832199,
                        0.019350704,
                        0.0015977627,
                        0.003447935,
                        0.017883845,
                        0.014826211,
                        -0.02617111,
                        -0.024304014,
                        -0.0041993647,
                        -0.020207208,
                        -0.008573829,
                        0.0033712676,
                        -0.006179999,
                        0.007474309,
                        0.0029519126,
                        -0.0014506467,
                        -0.013240714,
                        -0.016551293,
                        0.012099103,
                        -0.0050255316,
                        -0.0057253223,
                        -0.0021310174,
                        -0.008341446,
                        0.0041917237,
                        -0.008869906,
                        0.011595749,
                        -0.0016620221,
                        0.007877958,
                        0.008671747,
                        0.016892435,
                        0.008028487,
                        -0.02293363,
                        0.0087649925,
                        -0.03605261,
                        -0.013018509,
                        -0.011498792,
                        0.010009394,
                        0.010596368,
                        -0.0097576715,
                        0.007993852,
                        0.010651219,
                        0.0043800627,
                        -9.224701E-4,
                        0.040351115,
                        -0.0011116522,
                        -0.02665772,
                        0.013831533,
                        0.0020280543,
                        0.0116251325,
                        -0.032511443,
                        -0.0046799295,
                        0.008501904,
                        0.0048212362,
                        0.0026921472,
                        -0.0062492155
                    ]

Thanks @glund0 ! Run a few more experiments with a similar setup to what you described (i.e. similar to a single AWS r7a.xlarge data node, with 4 GB JVM heap and ~28 GB available for disk cache) but we weren’t able to reproduce these differences. There were some slight variations but nothing akin to this magnitude.

I wonder if you got the chance to experiment with the so_vector track setup? Monitor through JFR could also provide valuable information to help identify the reason for the slowness with index: false in your benchmarking scenario.

I created my own track with esrally to test my two vector cosine sim. I don’t see any difference in performance with my track between 8.19.3 and 9.1.3. The only thing I couldn’t test easily with esrally is using Java 24, which is what 9.1.3 comes bundled with. When I try to use Java 24 with esrally, it complains that it’s not a supported version. I’ll setup a standalone 9.1.3 install and have esrally use it just in case Java 24 is an issue. My assumption at this point is that there’s something wrong with our existing 9.1.3 setup. Comparing it to the esrally ES install should allow us to find the difference that’s causing the problem. Thanks for everyone’s help!

That’s really interesting. If you come across anything unexpected when using a specific set of params or env, we would really appreciate an update so that we can address any issues if they exist. Thanks!