# Data Transformation in elasticsearch

**URL:** <https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521>\
**Category:** Elasticsearch\
**Created:** [August 1, 2018, 9:26am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521 "2018-08-01T09:26:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [August 1, 2018, 9:26am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521/1 "2018-08-01T09:26:04Z")

</div>

Hi All,

We have been pulling data from multiple data sources and storing it in elasticsearch. Basically we use http\_poller method in logstash to pull in all the details.  
The problem we are facing is, all the data are stored in elasticsearch as string format, and while editing the field value i'm not getting integer field at all. Am i doing any mistakes here or should i need to convert all the values in logstash layer itself?

Screenshot FYI:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e4e1d07586e9bff2c7be9fbe983033e8d760ddf.png)

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 2, 2018, 9:42am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521/2 "2018-08-02T09:42:00Z")

</div>

You can change this in the logstash layer by using the mutate filter (see [https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-mutate.html#plugins-filters-mutate-convert](https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-mutate.html#plugins-filters-mutate-convert)), but you should also ensure in the elasticsearch mapping, that the correct types are specified. See [https://www.elastic.co/guide/en/elasticsearch/reference/6.3/mapping.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/mapping.html)

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [August 2, 2018, 10:01am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521/3 "2018-08-02T10:01:16Z")

</div>

@spinscale i tried using the mutate option and ended up with error,

**ERROR:**  
[2018-08-02T13:27:29,680][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 28, column 8 (byte 728) after filter\n {\n mutate { gsub =\> ["message", "\< ", "\<"] }\n mutate { gsub =\> ["message", "/ ", "/"] }\n xml {\n source =\> "message"\n store\_xml =\> true\n target =\> "theXML"\n force\_array =\> false\n }\nsplit\n {\n field =\> "[theXML][hoststatus]"\nmutate ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in`execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge\_state'"]}

Here is my config file:  
input {  
http\_poller {  
urls =\> {  
url =\> "[http://1.1.1.9/nagiosxi/backend/?cmd=gethoststatus&username=nagiosadmin&ticket=abcsyxz](http://1.1.1.9/nagiosxi/backend/?cmd=gethoststatus&username=nagiosadmin&ticket=abcsyxz)"  
}  
request\_timeout =\> 60  
schedule =\> { cron =\> "\* \* \* \* \*"}  
codec =\> "json"  
metadata\_target =\> "http\_poller\_metadata"  
}  
}  
filter  
{  
mutate { gsub =\> ["message", "\< ", "\<"] }  
mutate { gsub =\> ["message", "/ ", "/"] }  
xml {  
source =\> "message"  
store\_xml =\> true  
target =\> "theXML"  
force\_array =\> false  
}  
split  
{  
field =\> "[theXML][hoststatus]"  
mutate {convert =\> ["theXML.recordcount", "integer"]}  
mutate {convert =\> ["theXML.hoststatus.status\_update\_time", "date"]}  
mutate {convert =\> ["theXML.hoststatus.state\_type", "integer"]}  
mutate {convert =\> ["theXML.hoststatus.should\_be\_scheduled", "integer"]}  
}  
}  
output {  
stdout {  
codec =\> json  
}  
}

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 2, 2018, 11:38am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521/4 "2018-08-02T11:38:53Z")

</div>

you may want to ask this over in the logstash forum (my logstash skills are pretty limited).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2018, 11:38am UTC](https://discuss.elastic.co/t/data-transformation-in-elasticsearch/142521/5 "2018-08-30T11:38:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.