# Pinned query limit of 100 configurable? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 2, 2020, 3:48pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667 "2020-07-02T15:48:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 3:48pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/1 "2020-07-02T15:48:49Z")
How can I configure the maximum # of items returned in a 'pinned query'? I'm getting the following error: ```auto "caused_by": { "type": "x_content_parse_exception", "reason": "Failed to build [pinned] after last required field arrived", "caused_by": { "type": "illegal_argument_exception", "reason": "[pinned] Max of 100 ids exceeded: 121 provided." ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 4:11pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/2 "2020-07-02T16:11:38Z")
Hi Karl That's a fixed limit I'm afraid. Can you say any more about your use case where this would be a requirement? ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 4:26pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/3 "2020-07-02T16:26:33Z")
We're using this in conjunction with an external service and need to maintain the order of items being returned. We essentially have a list of products that we have to return to the client in the order we've received them. We wanted to leverage the same response format as our other search responses so this was by far the easiest way. Our items ID's are stored received as integers which makes scripting it a little more complex vs pinned queries we could just send as received. ```auto "pinned": { "ids": [ 146686, 2269502, 283749, 2730510, 318627, 931316, 997379, 1422420, 1644020, 1943821 ], ``` simply worked 🙂 Suggestions? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 4:30pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/4 "2020-07-02T16:30:28Z")
> [@Karl\_Kevilus](#): > > We essentially have a list of products that we have to return to the client in the order we've received them. Are you saying there's no "organic" element at all to these queries? ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 4:36pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/5 "2020-07-02T16:36:51Z")
For this use case, unfortunately, there is no 'organic' value. Just the order as provided in the array. Which is why the 'pinned query' seemed like a big win, until it wasn't. The expectation is that we could have as many as 1,000 items in the result set. That may be confusing.. essentially we send it like this: ```auto "query": { "bool": { "must": [ { "pinned": { "ids": [ 146686, 2269502, 283749, 2730510, 318627, 931316, 997379, 1422420, 1644020, 1943821 ], "organic": { "terms": { "product_id": [ 146686, 2269502, 283749, 2730510, 318627, 931316, 997379, 1422420, 1644020, 1943821 ] } } } } ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 4:40pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/6 "2020-07-02T16:40:13Z")
OK - obviously something outside of the index knows that `146686` is first and `2269502` is second and so on. Can't that information be recorded on the indexed docs in a "order` field and you just sort results by that field? Pinned queries are designed to blend "natural" and "unnatural" results where they are not the same set of docs. ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 4:51pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/7 "2020-07-02T16:51:00Z")
Totally understood about the design of Pinned queries. That's been my feedback to our team. The items we are returning are dynamically ordered by the external service and we have to maintain that order in the response. These 'ids' are both the document '\_id' and stored as a number 'product\_id' The 100 limit seems very low compared to any other ES default. Is there a major performance impact using pinned queries? Is a scripted score query a better option? \_MGET? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 5:01pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/8 "2020-07-02T17:01:47Z")
The [design](https://github.com/elastic/elasticsearch/issues/44074) of pinned queries was very focused. I'd recommend taking a look. It's not intended for serving long lists of documents in a particular order. > [@Karl\_Kevilus](#): > > The items we are returning are dynamically ordered by the external service and we have to maintain that order in the response. If the orders change infrequently there may be a case for updating an `order` field as I described. Otherwise, a simple `bool` query with a `should` array of `term` queries would suffice - each with a `boost` setting that reflects the intended order. Note however that queries containing thousands of query terms are to be avoided (which is partially why pinned query has a limit). Lucene has a default limit of 1024 clauses. Each search clause typically requires at least one random disk seek and also allocates a memory buffer to read lists of matching documents. These are not free resources, hence the limits. Storing an `order` field would be preferable if it's infrequently changed - the query should be much cheaper to run. ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 5:32pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/9 "2020-07-02T17:32:35Z")
What about something scripted like this vs boosting... just thinking about simplifying the code? Thoughts on performance? ```auto { "query": { "bool": { "must": [{ "terms": { "product_id": [377497, 931316, 931317, 127127] } }] } }, "sort": [{ "_script": { "type": "number", "script": { "lang": "painless", "inline": "if(params.products.containsKey(doc['_id'][0])) { return params.products[doc['_id'][0]];} return 9999;", "params": { "products": { "377497": 100, "931316": 200, "931317": 300 } } }, "order": "asc" } }] , "from": 0, "size": 10 } ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 5:38pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/10 "2020-07-02T17:38:42Z")
It's script and a doc value retrieval so likely slower than ``` bool should term product_id: 3777497, boost:100 term product_id: 931316, boost:200 ``` etc ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 5:39pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/11 "2020-07-02T17:39:11Z")
Great, thanks! ---
**Author:** ![Karl\_Kevilus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_kevilus/32/48470_2.png) [@Karl\_Kevilus](https://discuss.elastic.co/u/Karl_Kevilus)\ **Post date:** [July 2, 2020, 6:10pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/12 "2020-07-02T18:10:14Z")
A quick follow-up note for anyone having similar needs: I think the performance sorting these would be better on this if my product\_id's were stored as 'keyword' but since they are store as 'long' the performance is faster leveraging the painless script referencing the doc \_id's directly. With 400 products sorted in the query the painless script took ~22ms the should boost query ~90ms. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 2, 2020, 8:11pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/13 "2020-07-02T20:11:23Z")
Thanks for the benchmark! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 30, 2020, 8:11pm UTC](https://discuss.elastic.co/t/pinned-query-limit-of-100-configurable/239667/14 "2020-07-30T20:11:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.