Hi guys,
I'm very new to Elasticsearch stack, need some help over here...
Currently I'm trying to parse XML file, output to Elasticsearch and use it on Grafana for visualization. Now I facing a problem is my XML files are not parsing, not sure is my Logstash input pattern or/and the filter misconfigured.
Here are the resources:
XML format:
<log>
<entry>
<log_time>20230926-00:00:00</log_time>
<description><![CDATA[Connection established]]></description>
<service>FTP</service>
<sessionid>16692722</sessionid>
<type>0</type> <severity>0</severity>
<lstnconnaddr>x.x.x.156:21</lstnconnaddr>
<cliconnaddr>x.x.x.132:52948</cliconnaddr>
<cmd>start</cmd>
<sguid>C89C2B68-A602-4D43-6094-1CA63B6268A4</sguid>
</entry>
<entry>
<log_time>20230926-00:00:01</log_time>
<description><![CDATA[Connection established]]></description>
<service>FTP</service>
<sessionid>08919335</sessionid>
<type>0</type> <severity>0</severity>
<lstnconnaddr>x.x.x.156:21</lstnconnaddr>
<cliconnaddr>x.x.x.136:58003</cliconnaddr>
<cmd>start</cmd>
<sguid>C89C2B68-A602-4D43-6094-1CA63B6268A4</sguid>
</entry>
</log>
Logstash config file:
input {
file {
path => "/tmp/*.xml"
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_clean_after => 0.1
sincedb_write_interval => 30
type => "xml"
codec => multiline {
pattern => "^<log>"
negate => "true"
what => "previous"
}
file_sort_by => "last_modified"
}
}
filter {
xml {
source => "message"
target => "xmlLog"
store_xml => false
xpath => [
"/entry/log_time/text()", "log_time",
"/log/entry/description/text()", "description",
"/log/entry/service/text()", "service",
"/log/entry/sessionid/text()", "sessionid",
"/log/entry/type/text()", "type",
"/log/entry/severity/text()", "severity",
"/log/entry/cmd/text()", "cmd",
"/log/entry/sguid/text()", "sguid"
]
}
mutate {
gsub => [ "message", "\r\n", "" ]
}
mutate {
remove_field => ["@version","tags","_score", "_type", "type", "event", "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r", "\r\n", "host"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
}
stdout {
codec => rubydebug
}
}