XML Parsing incomplete

Hello everyone,

I'm sending xml logs into Logstash using Filebeat, it worked perfectly the first time I set up my configuration, but since I restarted Logstash service using "systemctl restart logstash.service" (I'm on CentOS7), the logs are entirely sent, but it seems that Logstash doesn't parse them completely.

Example of parsing error:

Here you can see the parsed message seen on Kibana:

And here the xml message which begin with "Alert message id" and not "Detect Time" like above.

My logstash conf:

input {
  beats {
    port => 5044
  }
}


filter 
  {
xml 
  {
    source => "message"
    store_xml => true
 	target => "parsed_data"
    xpath => 
		[
		
		"/Alert/Analyzer/Node/location/text()","Localisation",
		"/Alert/Analyzer/Node/name/text()","Nom",		
		"/Alert/Analyzer/Node/Address/address/text()","AdresseIP",
		"/Alert/Analyzer/Process/name/text()","Nom_manager",
		"/Alert/Analyzer/Process/pid/text()","pid",
		"/Alert/Analyzer/Process/path/text()","Chemin",
		"/Alert/Analyzer/Analyzer/name/text()","Nom_de_la_sonde",
		"/Alert/Analyzer/Analyzer/Node/address/text()","Adresse_sonde",
		"/Alert/Assessment/Impact/text()","Alerte",
		"/Alert/CreateTime/text()","Date"

		]

  }
}


output {
  elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

I think that because of this incomplete parsing, Kibana doesn't create appropriate fields (Localisation, Nom, AdresseIP,...):

Yet these fields were here before I restart Logstash service:

I hope you can help me to resolve my issue :slight_smile: . For my part, I did a lot of Internet searching but did not found a solution.

Regards,

Noe

I forgot to send an example of xml logs I'm parsing:

<IDMEF-Message>
  <Alert messageid="8543ca3c-a747-11ea-afd1">
    <Analyzer analyzerid="915065842660167" name="prelude-manager" manufacturer="http://www.prelude-siem.com" model="Prelude Manager" version="5.1.0" class="Concentrator" ostype="Linux" osversion="3.10.0-1062.18.1.el7.x86_64">
      <Node category="unknown">
        <location>Garidech</location>
        <name>SIEM IUT</name>
        <Address category="ipv4-addr">
          <address>127.0.0.1</address>
        </Address>
      </Node>
      <Process>
        <name>prelude-manager</name>
        <pid>4881</pid>
        <path>/usr/sbin/prelude-manager</path>
      </Process>
      <Analyzer analyzerid="1632282436192426" name="suricata" manufacturer="http://www.openinfosecfoundation.org/" model="Suricata" version="5.0.3" class="NIDS" ostype="Linux" osversion="4.18.0-147.8.1.el8_1.x86_64">
        <Node category="unknown">
          <location>Garidech</location>
          <name>Suricata</name>
          <Address category="ipv4-addr">
            <address>192.168.1.63</address>
          </Address>
        </Node>
        <Process>
          <name></name>
          <pid>2851</pid>
        </Process>
      </Analyzer>
    </Analyzer>
    <CreateTime ntpstamp="0xe284edb4.0xdae43000">2020-06-05T12:13:40.855044-04:00</CreateTime>
    <DetectTime ntpstamp="0xe284edb4.0xdad17000">2020-06-05T12:13:40.854758-04:00</DetectTime>
    <AnalyzerTime ntpstamp="0xe284edb4.0xdae5a000">2020-06-05T12:13:40.855066-04:00</AnalyzerTime>
    <Source spoofed="unknown">
      <Node category="unknown">
        <Address category="ipv4-addr">
          <address>192.168.1.63</address>
        </Address>
      </Node>
      <Service ip_version="4" iana_protocol_number="6" iana_protocol_name="tcp">
        <port>22</port>
      </Service>
    </Source>
    <Target decoy="unknown">
      <Node category="unknown">
        <Address category="ipv4-addr">
          <address>192.168.1.16</address>
        </Address>
      </Node>
      <Service ip_version="4" iana_protocol_number="6" iana_protocol_name="tcp">
        <port>52919</port>
      </Service>
    </Target>
    <Classification ident="1:20001" text="">
      <Reference origin="vendor-specific">
        <name>1:20001</name>
        <url>http://www.snort.org/search/sid/1-20001</url>
      </Reference>
    </Classification>
    <Assessment>
      <Impact severity="low" type="other">Connexion SSH</Impact>
    </Assessment>
    <AdditionalData type="string" meaning="proto_version">
      <string>2.0</string>
    </AdditionalData>
    <AdditionalData type="string" meaning="software_version">
      <string>OpenSSH_8.0</string>
    </AdditionalData>
    <AdditionalData type="string" meaning="proto_version">
      <string>2.0</string>
    </AdditionalData>
    <AdditionalData type="string" meaning="software_version">
      <string>OpenSSH_for_Windows_7.7</string>
    </AdditionalData>
    <AdditionalData type="integer" meaning="snort_rule_sid">
      <integer>20001</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="snort_rule_rev">
      <integer>0</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_ver">
      <integer>4</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_hlen">
      <integer>5</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_tos">
      <integer>0</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_len">
      <integer>124</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_id">
      <integer>49645</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_off">
      <integer>16384</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_ttl">
      <integer>64</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_proto">
      <integer>6</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="ip_sum">
      <integer>62702</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_seq">
      <integer>294683620</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_ack">
      <integer>3785619800</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_off">
      <integer>5</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_res">
      <integer>0</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_flags">
      <integer>24</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_win">
      <integer>251</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_sum">
      <integer>33806</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_urp">
      <integer>0</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_wscale">
      <integer>0</integer>
    </AdditionalData>
    <AdditionalData type="integer" meaning="tcp_hlen">
      <integer>20</integer>
    </AdditionalData>
    <AdditionalData type="byte-string" meaning="payload">
      <byte-string>8qRFpu2c/u40AEvUYz14PuPKTncKr/8fmP2hRBlNtsVJ3FFz/10kDCnGrJWCf/dOEuhnToESY6ZsOq6IMBR+T/jyJnCWUX6jL3u/TlFdzABaCt6a</byte-string>
    </AdditionalData>
  </Alert>
</IDMEF-Message>

Oh... I just add "IDMEF-Message" on my filter and it work...

filter 
  {
    xml 
      {
        source => "message"
        store_xml => true
 	target => "parsed_data"
	#force_array => true
        xpath => 
		[
		
		"IDMEF-Message/Alert/Analyzer/Node/location/text()","Localisation",
		"IDMEF-Message/Alert/Analyzer/Node/name/text()","Nom",		
		"IDMEF-Message/Alert/Analyzer/Node/Address/address/text()","AdresseIP",
		"IDMEF-Message/Alert/Analyzer/Process/name/text()","Nom_manager",
		"IDMEF-Message/Alert/Analyzer/Process/pid/text()","pid",
		"IDMEF-Message/Alert/Analyzer/Process/path/text()","Chemin",
		"IDMEF-Message/Alert/Analyzer/Analyzer/name/text()","Nom_de_la_sonde",
		"IDMEF-Message/Alert/Analyzer/Analyzer/Node/address/text()","Adresse_sonde",
		"IDMEF-Message/Alert/Assessment/Impact/text()","Alerte",
		"IDMEF-Message/Alert/CreateTime/text()","Date"

		]

      }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.