Logstash XML file not parsing

Elastic Stack Logstash
1

Hi guys,

I'm very new to Elasticsearch stack, need some help over here...
Currently I'm trying to parse XML file, output to Elasticsearch and use it on Grafana for visualization. Now I facing a problem is my XML files are not parsing, not sure is my Logstash input pattern or/and the filter misconfigured.

Here are the resources:
XML format:

image
image697×571 21.4 KB

<log>
  <entry>
    <log_time>20230926-00:00:00</log_time>
    <description><![CDATA[Connection established]]></description>
    <service>FTP</service>
    <sessionid>16692722</sessionid>
    <type>0</type>    <severity>0</severity>
    <lstnconnaddr>x.x.x.156:21</lstnconnaddr>
    <cliconnaddr>x.x.x.132:52948</cliconnaddr>
    <cmd>start</cmd>
    <sguid>C89C2B68-A602-4D43-6094-1CA63B6268A4</sguid>
  </entry>
  <entry>
    <log_time>20230926-00:00:01</log_time>
    <description><![CDATA[Connection established]]></description>
    <service>FTP</service>
    <sessionid>08919335</sessionid>
    <type>0</type>    <severity>0</severity>
    <lstnconnaddr>x.x.x.156:21</lstnconnaddr>
    <cliconnaddr>x.x.x.136:58003</cliconnaddr>
    <cmd>start</cmd>
    <sguid>C89C2B68-A602-4D43-6094-1CA63B6268A4</sguid>
  </entry>
</log>


Logstash config file:
input {
  file {
    path => "/tmp/*.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    sincedb_clean_after => 0.1
    sincedb_write_interval => 30
    type => "xml"
    codec => multiline {
      pattern => "^<log>"
      negate => "true"
      what => "previous"
    }
    file_sort_by => "last_modified"
  }
}

filter {
  xml {
    source => "message"
    target => "xmlLog"
    store_xml => false
    xpath => [
      "/entry/log_time/text()", "log_time",
      "/log/entry/description/text()", "description",
      "/log/entry/service/text()", "service",
      "/log/entry/sessionid/text()", "sessionid",
      "/log/entry/type/text()", "type",
      "/log/entry/severity/text()", "severity",
      "/log/entry/cmd/text()", "cmd",
      "/log/entry/sguid/text()", "sguid"
    ]
  }

  mutate {
    gsub => [ "message", "\r\n", "" ]
  }

  mutate {
    remove_field => ["@version","tags","_score", "_type", "type", "event", "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r", "\r\n", "host"]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => "elastic"
  }
  stdout {
    codec => rubydebug
  }
}
2

@Badger I saw you given a lot of advise of previous related issue, can you help me too?

3

Hi,

the root of the XML document is <log>, so you should start your XPath expressions with /log/entry/.

filter {
  xml {
    source => "message"
    target => "xmlLog"
    store_xml => false
    xpath => [
      "/log/entry/log_time/text()", "log_time",
      "/log/entry/description/text()", "description",
      "/log/entry/service/text()", "service",
      "/log/entry/sessionid/text()", "sessionid",
      "/log/entry/type/text()", "type",
      "/log/entry/severity/text()", "severity",
      "/log/entry/cmd/text()", "cmd",
      "/log/entry/sguid/text()", "sguid"
    ]
  }

Regards

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.