Convert string to numeric?

Elastic Stack Elasticsearch
1

I've got data being ingested into Elasticsearch which arrives in the incoming JSON as a string containing a floating point number, eg

"myfield": "1234.56"

So I've set a mapping for it

"myfield": { "type": "double" }

expecting then to be able to process it as numeric. And, indeed, after a mapping refresh, Kibana now believes this is a numeric field and appears to do numeric things with it, such as being able to display graphs of the value of this field. All good so far.

But if I do a search it comes back as a string

"_source": { "myfield": "1234.56" }

not as a number

"_source": { "myfield": 1234.56 }

which is what I would like for further processing of the search results in another language.

So what's going on here? If setting a mapping doesn't work, how do I get this field to be returned from a search as the correct type, if we take it as given that the software (several steps up the line) which generates the original JSON doesn't know about anything other than strings?

2

The _source that Elasticsearch returns is always the original document that you sent it. Even though Elasticsearch will internally treat that field as a double when running searches and aggregations, what gets returned as the _source is the original document.

What you need to do is clean up the documents before Elasticsearch indexes them. You could do so yourself before you send the documents to Elasticsearch, or you could use Elasticsearch' Ingest functionality (https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html). Ingest will be able to modify the source documents using a pipeline which is built out of processors. You could apply a convert processor to that field (https://www.elastic.co/guide/en/elasticsearch/reference/master/convert-processor.html) to convert it.

3

It turns out to be rather easier than I expected to correct the data type at source.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.