Import xml file in to Elasticsearch through logstash

Elastic Stack Logstash
1

I want to import xml file in to Elasticsearch through logstash.
I created config file as follows:
input
{
file
{
path => "D:/logstash-5.0.0/logstash-5.0.0/bin/data.xml"
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
xml {
source => "message"
}
}
output
{
elasticsearch
{
codec => json
hosts => "localhost"
index => "xmldata"
}
stdout
{
codec => rubydebug
}
}

I have created small data.xml file and the xml record is in one line only.
But I am getting following error.

D:\logstash-5.0.0\logstash-5.0.0\bin>logstash -f D:\logstash-5.0.0\logstash-5.0.0\bin\logstash-xmlConfig.conf
Sending Logstash logs to D:/logstash-5.0.0/logstash-5.0.0/logs which is now configured via log4j2.properties.
[2016-12-01T19:59:21,474][INFO
][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated
{:changes=>{:removed=>[], :added=>["http://localhost:9200"]}}
[2016-12-01T19:59:21,477][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2016-12-01T19:59:21,631][INFO
][logstash.outputs.elasticsearch] Attempting to install template
{:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"all"=>{"enabled"=>true, "norms"=>false}, "dynamictemplates"=>[{"message_field"=>{"path_match"=>"message",
"match_mapping_type"=>"string", "mapping"=>{"type"=>"text",
"norms"=>false}}}, {"string_fields"=>{"match"=>"",
"match_mapping_type"=>"string", "mapping"=>{"type"=>"text",
"norms"=>false,
"fields"=>{"keyword"=>{"type"=>"keyword"}}}}}],
"properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword",
"include_in_all"=>false}, "geoip"=>{"dynamic"=>true,
"properties"=>{"ip"=>{"type"=>"ip"},
"location"=>{"type"=>"geo_point"},
"latitude"=>{"type"=>"half_float"},
"longitude"=>{"type"=>"half_float"}}}}}}}}
[2016-12-01T19:59:21,638][INFO
][logstash.outputs.elasticsearch] New Elasticsearch output
{:class=>"LogStash::Outputs::ElasticSearch",
:hosts=>["localhost"]}
[2016-12-01T19:59:23,329][ERROR][logstash.agent
] Pipeline aborted due to error {:exception=>#,
:backtrace=>["D:/logstash-5.0.0/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.1/lib/logstash/filters/xml.rb:106:in
register'", "D:/logstash-5.0.0/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:197:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "D:/logstash-5.0.0/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:197:instart_workers'", "D:/logstash-5.0.0/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:153:in run'", "D:/logstash-5.0.0/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:250:instart_pipeline'"]}
[2016-12-01T19:59:23,455][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2016-12-01T19:59:26,354][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

I am using 5.x version

2

This part of the log:

Pipeline aborted due to error {:exception=>#, 
:backtrace=>["D:/logstash-5.0.0/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.1/lib/logstash/filters/xml.rb:106:in

Doesn't it actually include an error message too? Something like

Pipeline aborted due to error {:exception=>#<When the 'store_xml' configuration option is true, 'target' must also be set>, 
:backtrace=>["D:/logstash-5.0.0/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.1/lib/logstash/filters/xml.rb:106:in

? Either way, the interesting error message is:

When the 'store_xml' configuration option is true, 'target' must also be set

3

actual error log was:-

LogStash::ConfigurationError: translation missing: en.logstash.agent.configuration.invalid_plugin_register

The above error with logstash 5.0.0
As per logstash 5.0.1 release note above error was resolved in 5.0.1 version
https://www.elastic.co/guide/en/logstash/current/logstash-5-0-1.html

Now I am using 5.0.2 version and getting below error with same conf and data file

ipeline aborted due to error {:exception=>#LogStash::ConfigurationError: Cannot register filter xml plugin. The error reported is: When the 'store_xml' configuration option is true, 'target' must also be set>,

Please correct me If I doing anything wrong

4

The xml filter is complaining about you not setting the target option.

1 Like
5

Thanks Magnus,
After setting target option it is working fine

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.