Data Transformation in elasticsearch

Elastic Stack Elasticsearch
1

Hi All,

We have been pulling data from multiple data sources and storing it in elasticsearch. Basically we use http_poller method in logstash to pull in all the details.
The problem we are facing is, all the data are stored in elasticsearch as string format, and while editing the field value i'm not getting integer field at all. Am i doing any mistakes here or should i need to convert all the values in logstash layer itself?

Screenshot FYI:

image
image.png979×336 2.78 KB

Thanks
Gauti

2

You can change this in the logstash layer by using the mutate filter (see https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-mutate.html#plugins-filters-mutate-convert), but you should also ensure in the elasticsearch mapping, that the correct types are specified. See https://www.elastic.co/guide/en/elasticsearch/reference/6.3/mapping.html

3

@spinscale i tried using the mutate option and ended up with error,

ERROR:
[2018-08-02T13:27:29,680][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 28, column 8 (byte 728) after filter\n {\n mutate { gsub => [ "message", "< ", "<" ] }\n mutate { gsub => [ "message", "/ ", "/" ] }\n xml {\n source => "message"\n store_xml => true\n target => "theXML"\n force_array => false\n }\nsplit\n {\n field => "[theXML][hoststatus]"\nmutate ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

Here is my config file:
input {
http_poller {
urls => {
url => "http://1.1.1.9/nagiosxi/backend/?cmd=gethoststatus&username=nagiosadmin&ticket=abcsyxz"
}
request_timeout => 60
schedule => { cron => "* * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter
{
mutate { gsub => [ "message", "< ", "<" ] }
mutate { gsub => [ "message", "/ ", "/" ] }
xml {
source => "message"
store_xml => true
target => "theXML"
force_array => false
}
split
{
field => "[theXML][hoststatus]"
mutate {convert => ["theXML.recordcount", "integer"]}
mutate {convert => ["theXML.hoststatus.status_update_time", "date"]}
mutate {convert => ["theXML.hoststatus.state_type", "integer"]}
mutate {convert => ["theXML.hoststatus.should_be_scheduled", "integer"]}
}
}
output {
stdout {
codec => json
}
}

Thanks
Gauti

4

you may want to ask this over in the logstash forum (my logstash skills are pretty limited).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.