# Script Sorting Appers to be Limited to 32 bit floats

**URL:** <https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469>\
**Category:** Elasticsearch\
**Created:** [July 19, 2012, 6:34pm UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469 "2012-07-19T18:34:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [July 19, 2012, 6:34pm UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/1 "2012-07-19T18:34:26Z")

</div>

I'm using ElasticSearch Version 0.17.6

Currently, I'm running a Java Sort Script 3 Letters Plus Number String Sort[https://gist.github.com/3145802](https://gist.github.com/3145802) on  
one field.

I need to extend this to cover multi-column sort, unfortunately  
ElasticSearch (in my version) seems limited to me creating numbers that  
only differ from each other by about 24 or 30 bits, no matter whether I  
implement AbstractFloatSearchScript, AbstractLongSearchScript  
or AbstractDoubleSearchScript - I thought that Long and Double might give  
me 64 bits worth of room to compare numbers with - but no matter which  
version I use I always appear to run into a limit where that if numbers  
differ by more than about 2^30 - then ElasticSearch seems to refuse to  
compare them.

Difficult to do a reproduction, this is my attempt to expand the range of  
numbers I compare to 64 bits (I also tried using Doubles as well as Longs) Attempted  
Multi-Colum Sort Using Long return value. [https://gist.github.com/3145858](https://gist.github.com/3145858)

David.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 20, 2012, 8:19am UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/2 "2012-07-20T08:19:04Z")

</div>

Hi David

> Currently, I'm running a Java Sort Script 3 Letters Plus Number String  
> Sort on one field.

> I need to extend this to cover multi-column sort, unfortunately  
> Elasticsearch (in my version) seems limited to me creating numbers  
> that only differ from each other by about 24 or 30 bits, no matter  
> whether I implement AbstractFloatSearchScript,  
> AbstractLongSearchScript or AbstractDoubleSearchScript - I thought  
> that Long and Double might give me 64 bits worth of room to compare  
> numbers with - but no matter which version I use I always appear to  
> run into a limit where that if numbers differ by more than about 2^30
> 
> - then Elasticsearch seems to refuse to compare them.

I'll leave the Java internals to somebody else, but using scripts for  
sorting is not the most efficient way of doing it, not least because the  
results don't get cached.

Can you give us an example of real data and how you want to sort it? It  
may be possible to index the data in a way that you can sort without  
scripts.

ta

clint

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [July 20, 2012, 9:12am UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/3 "2012-07-20T09:12:40Z")

</div>

Hi Clinton,

From your response, are you saying that custom scripts are limited to 32  
bits only. I was hoping that someone would know how to  
extend AbstractSearchScript, AbstractExecutableScript to give me more  
numbers to sort against.

I kind of anticipated that someone would suggest altering the way I index  
this particular field, and that is a reasonable question, but the reasons  
for using what we do, the way we do it, is quite involved.

The data structure we are using is a field on a contact, the field is  
defined like this:

"customColumns": {  
"type":"nested",  
"properties" : {  
"ccId" : {"type" : "string", "index" : "not\_analyzed"},  
"value" : {"type" : "string"},  
"valueLC" : {"type" : "string", "index" : "not\_analyzed"}  
}  
}

It is essentially a name:value mapping between ccId and value. We use  
calculate this ccId at runtime to create "context aware fields". Depending  
on the userId and the list that a contact is in a ccId is generated at  
runtime. Then, at runtime, the value of the ccId is looked up in the  
customColumns source (which is a map in the java) and added as a top-level  
field on the contact (which also is a map). In pseudo code the process,  
which is executed at runtime, after the query results are returned is:

ccId, ccName = f (contactId, userId, listId);

Map contactSource = results.getSource();

Map customColsSource = contactSource["customColumns"];

value = customColsSource["ccId"];

contactSource[ccName] = value;

delete contactSource["customColumns"]

... then convert List to JSON and return.

The effect of this is to create a context aware field on the index. Each  
user my have up to 50 of these, and there are 3000 users, so we potentially  
have something like 150,000 fields that can be added to a contact depending  
on context.

Otherwise, contacts have about 70 fields. 400,000 contacts take up about  
1.5GB of VM memory.

- David.

On Friday, July 20, 2012 9:19:04 AM UTC+1, Clinton Gormley wrote:

> Hi David
> 
> > Currently, I'm running a Java Sort Script 3 Letters Plus Number String  
> > Sort on one field.
> 
> > I need to extend this to cover multi-column sort, unfortunately  
> > Elasticsearch (in my version) seems limited to me creating numbers  
> > that only differ from each other by about 24 or 30 bits, no matter  
> > whether I implement AbstractFloatSearchScript,  
> > AbstractLongSearchScript or AbstractDoubleSearchScript - I thought  
> > that Long and Double might give me 64 bits worth of room to compare  
> > numbers with - but no matter which version I use I always appear to  
> > run into a limit where that if numbers differ by more than about 2^30
> > 
> > - then Elasticsearch seems to refuse to compare them.
> 
> I'll leave the Java internals to somebody else, but using scripts for  
> sorting is not the most efficient way of doing it, not least because the  
> results don't get cached.
> 
> Can you give us an example of real data and how you want to sort it? It  
> may be possible to index the data in a way that you can sort without  
> scripts.
> 
> ta
> 
> clint

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 20, 2012, 9:23am UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/4 "2012-07-20T09:23:47Z")

</div>

Hiya

> From your response, are you saying that custom scripts are limited to  
> 32 bits only. I was hoping that someone would know how to extend  
> AbstractSearchScript, AbstractExecutableScript to give me more numbers  
> to sort against.

No I'm not - I have no idea 🙂

> The effect of this is to create a context aware field on the index.  
> Each user my have up to 50 of these, and there are 3000 users, so we  
> potentially have something like 150,000 fields that can be added to a  
> contact depending on context.

OK, so my solution seems unlikely to help

clint

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [July 20, 2012, 10:13am UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/5 "2012-07-20T10:13:59Z")

</div>

Thanks for your input Clinton, much appreciated, as always.

Regarding the sort scripts, I've been looking at the JavaCode that invokes  
the custom sort scripts, it is a class called CustomScoreQueryParser that  
has an inner class that invokes this function:

@Override public float score(int docId, float subQueryScore) {  
script.setNextDocId(docId);  
script.setNextScore(subQueryScore);  
return script.runAsFloat();  
}

I think the particular line is \*\*\*\* return script.runAsFloat() \*\*\*\*\*.

It looks like the Lucene Score collector (TopScoreDocCollector) is  
expecting floats, when invoking this code:

public void collect(int doc) throws IOException {  
float score = scorer.score();

so that is where the limitation lies - the Lucene Collector expects  
floats, so the Elasticsearch custom script has no option but to give it  
floats, even if the calculation is done in terms of Longs or Doubles, it  
will always be cast to a float.

Seems like I have no way round this limitation.

If there are any other suggestions I would welcome them.

- David.

On Friday, July 20, 2012 10:23:47 AM UTC+1, Clinton Gormley wrote:

> Hiya
> 
> > From your response, are you saying that custom scripts are limited to  
> > 32 bits only. I was hoping that someone would know how to extend  
> > AbstractSearchScript, AbstractExecutableScript to give me more numbers  
> > to sort against.
> 
> No I'm not - I have no idea 🙂
> 
> > The effect of this is to create a context aware field on the index.  
> > Each user my have up to 50 of these, and there are 3000 users, so we  
> > potentially have something like 150,000 fields that can be added to a  
> > contact depending on context.
> 
> OK, so my solution seems unlikely to help
> 
> clint

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/script-sorting-appers-to-be-limited-to-32-bit-floats/8469/6 "2017-07-06T03:19:39Z")

</div>

Closed on July 6, 2017, 3:19am UTC