Double datatypes values returned as 0 rather than 0.0 in Elastic Search Results

Elastic Stack Elasticsearch
1

Hi
We are in a process of migrating ES from 1.5.2 to 6.1.1 and we hit a problem.

We have an index where one of the properties is of type "double".
The value for the property in the documents loaded into the index is 0.0.

Unfortunately when i query the index the results of the document is returned as 0, which is breaking my client code whose data type is autosuggested to long rather than double.

where as 1.5.2 version is returning 0.0.

Please suggest a solution to fix this issue

2

Can you show your mapping?

GET your-index/_mapping
3 
			{
			   "parent_code": {
				  "mappings": {
					 "doc": {
						"dynamic_templates": [
						   {
							  "strings_as_keywords": {
								 "match_mapping_type": "string",
								 "mapping": {
									"type": "keyword"
								 }
							  }
						   },
						   {
							  "row_keys_datatype": {
								 "match": "ROWKEY",
								 "mapping": {
									"type": "long"
								 }
							  }
						   }
						],
						"properties": {
						   "@timestamp": {
							  "type": "date"
						   },
						   "@version": {
							  "type": "keyword"
						   },
						   "BOTTOM_DECILE": {
							  "type": "double"
						   },
						   "BenchMark10": {
							  "type": "float"
						   },
						   "BenchMark3": {
							  "type": "float"
						   },
						   "BenchMark4": {
							  "type": "float"
						   },
						   "BenchMark5": {
							  "type": "float"
						   },
						   "BenchMark6": {
							  "type": "float"
						   },
						   "BenchMark7": {
							  "type": "float"
						   },
						   "BenchMark8": {
							  "type": "float"
						   },
						   "BenchMark9": {
							  "type": "float"
						   }
						   "EARNED_SCORE_BONUS_POINT": {
							  "type": "float"
						   },
						   "HAS_SUB_MEASURES": {
							  "type": "long"
						   },
						   "HIGH_PERFORMANCE": {
							  "type": "long"
						   }
							
						   "PAGE_ID": {
							  "type": "long"
						   },
						   "id": {
							  "type": "keyword"
						   }
						}
					 }
				  }
			   }
			}

I am referring to EARNED_SCORE_BONUS_POINT field

4

Are you sure that you're sending "0.0" in the document source ? Can you show a sample document which contains "0.0" in its source, but "0" comes back in the results ?

5

Below are links for actual Mappings, Source and results
Mapping
Doc Source loaded from logstash
Results

6

The mapping you linked to is defined in the parent_measure_code index but the document you're showing comes from the parent_code index. Can you show the mapping from your parent_code index, too?

7

its indeed parent_measure_code, I changed the index name and deleted few unwanted fields in my first post. The information I shared in the links are the actual ones

8

The source you're seeing (3rd link) is really exactly what you have sent to ES. ES does never modify the source document. So we need more insights into how your indexing process looks like. Does that happen via Logstash?

9

Yes, below is the logstash Conf file link
Logstash

And the Db query always returns the results as 0.0 ,for the field in question

If you observe the "EARNED_SCORE_BONUS_POINT" fields they have different values in both source and results

10

Hi, Is there a way at least to override the default behavior and return the values honoring the datatypes of the fields. May be any plugin where i can transform the json response.

Its quite urgent.

Thank you.

11

One thing you might want to try is to send your field as a string (but keep the float type in the mapping !!) and thus the value will be unaltered when it comes back

In your Logstash config you need to add a mutate/convert filter

filter {
   mutate {
      convert => { "EARNED_SCORE_BONUS_POINT" => "string" }
   }
}

Then you keep your mapping as it is:

PUT parent_measure_code
{
  "mappings": {
    "doc": {
      "properties": {
        "EARNED_SCORE_BONUS_POINT": {
          "type": "float"
        }
      }
    }
  }
}

You index the document

PUT parent_measure_code/doc/1
{
  "EARNED_SCORE_BONUS_POINT": "0.0"
}

When retrieving it will get "0.0" (as a string though):

GET parent_measure_code/doc/1
{
  "EARNED_SCORE_BONUS_POINT": "0.0"
}

Not sure if that helps...

12

Thanks for the information.
But It won't help, as there are many fields (float) whose values are 0.0 being returned as 0 across multiple docs.

I'm looking for a fix at core level by sending the appropriate json value based on the field data type, so that the json result is mapped correctly at client end.

If you any ideas to change the behavior in base code or via plugins, pls don't mind dropping drop it at niranjan.518@gmail.com. As said earlier, I'm in a bad need of the fix.

13

On the other hand, I'm curious to know why your client side code can't cope with 0 instead of 0.0... Seems somewhat fragile...

14

The ObjectMapper class from jackson library auto detects the java datatype based on the json value from ES Results and creates the HashMap of the fields from json. So now 2 documents from same index with values of 0 and 0.0 for a given field will create 2 objects with different datatypes, Integer and Double. Which is wrong. We explicitly dont provide data type mapping while tranforming results, but rather rely on ES fields data types.

15

You might want to look towards writing a custom deserializer for this.

See https://stackoverflow.com/a/7334405/4604579

16

Hi, do you see this as a bug in the base code of Elastic Search ?

17

After debugging the elastic search code, I found that the issue is with logstash elastic search plugin. ES never saw 0.0 in the source. It is the plugin that is converting 0.0 to 0 while putting the doc into the ES, but stdout prints 0.0 to console. Im still wondering about the odd behavior of the plugin to strip the .0

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.