BigDecimal support

Elastic Stack Elasticsearch
1

In financial services space, we almost never use float/double in our domain

  • we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a lossless
manner (ie what I get back from _source has the same precision, etc as what
I put in).

When I have had to preserve the json serialisation of BigDecimal, I have
usually had custom serialiser/deserialisers that printed it out as a json
number - but whose textual value was toPlainString(). When deserialising,
creating the BigDecimal with the string value (e.g. '42.5400') maintained
the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in the
source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

ES accepts BigDecimal input. You can specify scale and rounding mode to
format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if the
number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in any
case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky nick.minutello@gmail.com wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I have
usually had custom serialiser/deserialisers that printed it out as a json
number - but whose textual value was toPlainString(). When deserialising,
creating the BigDecimal with the string value (e.g. '42.5400') maintained
the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in the
source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBKpPHFN%3DoFFka%3Dk%3Dtk%3DOLmSqB9kbY0RSOC0nM4C5Lww%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

XContentBuilder has support for BigDecimal, but:

  1. If you pass the source as a Map when indexing, the BigDecimal
    handling doesn't get invoked
    (IndexRequestBuilder.setSource(Map<String, Object> source) does not handle BigDecimal · Issue #5260 · elastic/elasticsearch · GitHub).
  2. The existing handling should delegate through to Jackson's handling
    of BigDecimal (which can be configured to serialise BigDecimal in a
    lossless fashion - I dont think that feature existed when I had to worry
    about it last)

Looking at the code now, I think its actually an easy change - I will see
if I can create a pull request.

-Nick

On Wednesday, 26 February 2014 17:28:29 UTC, Jörg Prante wrote:

ES accepts BigDecimal input. You can specify scale and rounding mode to
format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if the
number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in any
case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky <nick.mi...@gmail.com <javascript:>

wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I have
usually had custom serialiser/deserialisers that printed it out as a json
number - but whose textual value was toPlainString(). When deserialising,
creating the BigDecimal with the string value (e.g. '42.5400') maintained
the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in the
source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/418b171f-ea20-4416-8324-d7b3b5d113f7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I am ready to create a pull request - its actually quite a simple change.
However, I cant find *any *tests for the existing BigDecimal support ...
does that sound right?

-Nick

On Friday, 28 February 2014 12:09:00 UTC, mooky wrote:

XContentBuilder has support for BigDecimal, but:

  1. If you pass the source as a Map when indexing, the BigDecimal
    handling doesn't get invoked (
    IndexRequestBuilder.setSource(Map<String, Object> source) does not handle BigDecimal · Issue #5260 · elastic/elasticsearch · GitHub).
  2. The existing handling should delegate through to Jackson's handling
    of BigDecimal (which can be configured to serialise BigDecimal in a
    lossless fashion - I dont think that feature existed when I had to worry
    about it last)

Looking at the code now, I think its actually an easy change - I will see
if I can create a pull request.

-Nick

On Wednesday, 26 February 2014 17:28:29 UTC, Jörg Prante wrote:

ES accepts BigDecimal input. You can specify scale and rounding mode to
format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if the
number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in any
case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull
request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky nick.mi...@gmail.com wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I have
usually had custom serialiser/deserialisers that printed it out as a json
number - but whose textual value was toPlainString(). When deserialising,
creating the BigDecimal with the string value (e.g. '42.5400') maintained
the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in the
source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Yes, there are no tests yet.

Jörg

On Wed, Mar 5, 2014 at 2:24 PM, mooky nick.minutello@gmail.com wrote:

I am ready to create a pull request - its actually quite a simple change.
However, I cant find *any *tests for the existing BigDecimal support ...
does that sound right?

-Nick

On Friday, 28 February 2014 12:09:00 UTC, mooky wrote:

XContentBuilder has support for BigDecimal, but:

  1. If you pass the source as a Map when indexing, the BigDecimal
    handling doesn't get invoked (https://github.com/
    elasticsearch/elasticsearch/issues/5260https://github.com/elasticsearch/elasticsearch/issues/5260
    ).
  2. The existing handling should delegate through to Jackson's
    handling of BigDecimal (which can be configured to serialise BigDecimal in
    a lossless fashion - I dont think that feature existed when I had to worry
    about it last)

Looking at the code now, I think its actually an easy change - I will see
if I can create a pull request.

-Nick

On Wednesday, 26 February 2014 17:28:29 UTC, Jörg Prante wrote:

ES accepts BigDecimal input. You can specify scale and rounding mode to
format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/
8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if
the number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in any
case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull
request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky nick.mi...@gmail.com wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I
have usually had custom serialiser/deserialisers that printed it out as a
json number - but whose textual value was toPlainString(). When
deserialising, creating the BigDecimal with the string value (e.g.
'42.5400') maintained the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in
the source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGYL7-_ocoZNq%2Bv71z-o6Jisi3J_8mVmsqU1UJuzWjPog%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Righto - I will try add some.
-Nick

On Wednesday, 5 March 2014 13:48:58 UTC, Jörg Prante wrote:

Yes, there are no tests yet.

Jörg

On Wed, Mar 5, 2014 at 2:24 PM, mooky <nick.mi...@gmail.com <javascript:>>wrote:

I am ready to create a pull request - its actually quite a simple change.
However, I cant find *any *tests for the existing BigDecimal support ...
does that sound right?

-Nick

On Friday, 28 February 2014 12:09:00 UTC, mooky wrote:

XContentBuilder has support for BigDecimal, but:

  1. If you pass the source as a Map when indexing, the BigDecimal
    handling doesn't get invoked (https://github.com/
    elasticsearch/elasticsearch/issues/5260https://github.com/elasticsearch/elasticsearch/issues/5260
    ).
  2. The existing handling should delegate through to Jackson's
    handling of BigDecimal (which can be configured to serialise BigDecimal in
    a lossless fashion - I dont think that feature existed when I had to worry
    about it last)

Looking at the code now, I think its actually an easy change - I will
see if I can create a pull request.

-Nick

On Wednesday, 26 February 2014 17:28:29 UTC, Jörg Prante wrote:

ES accepts BigDecimal input. You can specify scale and rounding mode to
format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/
8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if
the number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in
any case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull
request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky nick.mi...@gmail.com wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I
have usually had custom serialiser/deserialisers that printed it out as a
json number - but whose textual value was toPlainString(). When
deserialising, creating the BigDecimal with the string value (e.g.
'42.5400') maintained the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in
the source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d56144d1-3e0d-46d2-9ff8-a2cadc9b8344%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Trying to submit a pull request. Getting a 403 ....
-Nick

On Monday, 10 March 2014 17:28:24 UTC, mooky wrote:

Righto - I will try add some.
-Nick

On Wednesday, 5 March 2014 13:48:58 UTC, Jörg Prante wrote:

Yes, there are no tests yet.

Jörg

On Wed, Mar 5, 2014 at 2:24 PM, mooky nick.mi...@gmail.com wrote:

I am ready to create a pull request - its actually quite a simple change.
However, I cant find *any *tests for the existing BigDecimal support
... does that sound right?

-Nick

On Friday, 28 February 2014 12:09:00 UTC, mooky wrote:

XContentBuilder has support for BigDecimal, but:

  1. If you pass the source as a Map when indexing, the BigDecimal
    handling doesn't get invoked (https://github.com/
    elasticsearch/elasticsearch/issues/5260https://github.com/elasticsearch/elasticsearch/issues/5260
    ).
  2. The existing handling should delegate through to Jackson's
    handling of BigDecimal (which can be configured to serialise BigDecimal in
    a lossless fashion - I dont think that feature existed when I had to worry
    about it last)

Looking at the code now, I think its actually an easy change - I will
see if I can create a pull request.

-Nick

On Wednesday, 26 February 2014 17:28:29 UTC, Jörg Prante wrote:

ES accepts BigDecimal input. You can specify scale and rounding mode
to format the BigDecimal.

https://github.com/jprante/elasticsearch/commit/
8ef8cd149b867e3e45bc3055dfd6da80e4e9c7ec

Internally, BigDecimal is automatically converted to a JSON string if
the number does not fit into double format. Because numbers are useful in
Lucene for range searches, they have an advantage.

But I agree, another option could be to enforce string conversion in
any case, for example storing currency values as strings for financial
services, without arithmetic operations in the index.

Maybe the toEngineeringString() was not a smart decision and
toPlainString() works better.

So I would welcome improvements, or should I suggest one in a pull
request?

Jörg

On Wed, Feb 26, 2014 at 6:05 PM, mooky nick.mi...@gmail.com wrote:

In financial services space, we almost never use float/double in our
domain - we always use BigDecimal.

In elastic, I would like to be able to index/store BigDecimal in a
lossless manner (ie what I get back from _source has the same precision,
etc as what I put in).

When I have had to preserve the json serialisation of BigDecimal, I
have usually had custom serialiser/deserialisers that printed it out as a
json number - but whose textual value was toPlainString(). When
deserialising, creating the BigDecimal with the string value (e.g.
'42.5400') maintained the precision that was originally serialised
e.g.

{
verySmallNumber : 0.00000000012000,
otherNumber : 42.5400
}

Perhaps elastic could index bigdecimal as a double - but store it in
the source in a lossless fashion.
It would require a user setting, I guess, to treat all floating point
numbers as BigDecimal.

Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b54dfd5a-3a0e-4946-aa5f-28b3794a92ac%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b8463a21-c997-4269-ae52-992caae88ced%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f567a7fe-7c41-438d-95a5-3c7415f6b32b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Fixed. Pull request
here: https://github.com/elasticsearch/elasticsearch/pull/5491

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/84755e75-13f2-4f80-894f-b88452a8de68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

There is a more complete pull request
here: BigInteger/BigDecimal support by jprante · Pull Request #5683 · elastic/elasticsearch · GitHub

On Saturday, 22 March 2014 00:31:33 UTC, mooky wrote:

Fixed. Pull request here:
Add support for lossless storage of BigDecimal numeric values in _source by nickminutello · Pull Request #5491 · elastic/elasticsearch · GitHub

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/84dad4e7-4935-4349-99c6-a0d3c355c7b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.