GeoPoint precision

(Yevhenii Kurtov) #1


Today we got a strange failure in our application tests : app wrote 10.783551695140882 but got 10.783551695140883 in reply.
I know that everything after eight number after decimal point doesn't make sense, but it'll be very inconvenient to deal with random results in the tests.

(Michael McCandless) #2

This is par for the course with floating point numbers.

That number you stored (10.783551695140882) cannot be accurately represented as an IEEE double precision number (which is how Java/Lucene/Elasticsearch encode full precision floats): that last 2 turning into a 3 is the closest double that can be represented.

The truncation gets even worse if you use 4 byte and 2 byte (half) floats.

Mike McCandless

(Yevhenii Kurtov) #3

Thank you @mikemccand, we truncated number till 8 number after decimal point as an easiest option.

(system) #4