Date parse failure

Elastic Stack Logstash
1

Hello,

I struggle to understand why my date matching is failing.
We trying to build a reporting application using elasticsearch for our Storage.

I've that kind of data in an XML:

<Event>
  <System>
    <EventID>0</EventID>
    <EventName>Open Object</EventName>
    <Source>CIFS</Source>
    <Result>Operation Success</Result>
    <TimeCreated SystemTime="2020-09-23 00:09:20"/>
    <Computer>d5c2b245-cafa-ea11-80d0-005056923059</Computer>
  </System>
  <EventData>
    <ClientIP IPVersion="4">10.244.243.161</ClientIP>
    <UnixID Uid="2930071385" Gid="2930000513"></UnixID>
    <WindowsSID>S-1-5-21-1141645092-1946699681-618671499-71385</WindowsSID>
    <UserIsLocal>0</UserIsLocal>
    <DomainName>STLUC</DomainName>
    <UserName>SP_P_SearchContent</UserName>
    <ObjectType>File</ObjectType>
    <HandleID>15397808166</HandleID>
    <ObjectName>(service);/info/library/Software UCL/Archives/BigFive/SAS-9.4.TS1M5/product_data/base__94110_cd563__wx6__pt__sp0__1/pt_nls.xml</ObjectName>
    <UserACE>0</UserACE>
    <UserACEMask>1</UserACEMask>
    <DesiredAccess>1179785</DesiredAccess>
    <Attributes>128</Attributes>
  </EventData>
</Event>

Everything is ok with the following logstash configuration:

input {
   beats { port => 5044 }
}
 
filter {
    xml {
       source => "message"
       xpath => [
                 "/Event/System/EventID/text()", "EventID",
                 "/Event/System/EventName/text()", "EventName",
                 "/Event/System/Source/text()", "Source",
                 "/Event/System/Result/text()", "Result",
                 "/Event/System/TimeCreated/@SystemTime", "SystemTime",
                 "/Event/System/Computer/text()", "Computer",
                 "/Event/EventData/ClientIP/text()", "ClientIP",
                 "/Event/EventData/UnixID/text()", "UnixID",
                 "/Event/EventData/WindowsSID/text()", "WindowsSID",
                 "/Event/EventData/UserIsLocal/text()", "UserIsLocal",
                 "/Event/EventData/DomainName/text()", "DomainName",
                 "/Event/EventData/UserName/text()", "UserName",
                 "/Event/EventData/ObjectType/text()", "ObjectType",
                 "/Event/EventData/HandleID/text()", "HandleID",
                 "/Event/EventData/ObjectName/text()", "ObjectName",
                 "/Event/EventData/ReadOffSet/text()", "ReadOffset",
                 "/Event/EventData/ReadCount/text()", "ReadCount",
                 "/Event/EventData/UserACE/text()", "UserACE",
                 "/Event/EventData/UserACEMask/text()", "UserACEMask",
                 "/Event/EventData/DesiredAcces/text()", "DesiredAcces",
                 "/Event/EventData/Attributes/text()", "Attributes",
                 "/Event/EventData/WriteOffSet/text()", "WriteOffset",
                 "/Event/EventData/WriteCount/text()", "WriteCount",
                 "/Event/EventData/InformationSet/text()", "InformationSet",
                 "/Event/EventData/NewDirHandle/text()", "NewDirHandle",
                 "/Event/EventData/NewPath/text()", "NewPath",
                 "/Event/EventData/ACECount/text()", "ACECount",
                 "/Event/EventData/FailureReason/text()", "FailureReason"
       ]
       store_xml => true
       target => "doc"
    }

    date {
      match => ["SystemTime", "yyyy-MM-dd HH:mm:ss"]
          timezone => "Europe/Brussels"
          target => "@timestamp"
    }
}

The SystemTime field is correctly populated with the date value but the document on elasticsearch have the tags _dateparsefailure and the timestamp is not replaced.

Is someone have any clue ? It really seems to me that I'm right in my logstash conf.

2

Hi @Wilfried

Yup so first in logstash when debugging you should always try debug output in the output section

stdout { codec => rubydebug }

If you did you would see the output is put into arrays example

"System" => [
            [0] {
                "TimeCreated" => [
                    [0] {
                        "SystemTime" => "2020-09-23 00:09:20"
                    }
                ],

So what you need to is assuming you want to is not put all the data into arrays.
There is a force_array parameter

filter {
    xml {
       source => "message"
       force_array => false
...
    date {
      match => ["SystemTime", "yyyy-MM-dd HH:mm:ss"]
          timezone => "Europe/Brussels"
          target => "@timestamp"
    }

And I think you see what you want...

       "ClientIP" => "10.244.243.161",
        "UserACE" => "0",
       "UserName" => "SP_P_SearchContent",
     "ObjectType" => "File",
    "UserACEMask" => "1",
     "@timestamp" => 2020-09-22T22:09:20.000Z,
     "ObjectName" => "(service);/info/library/Software UCL/Archives/BigFive/SAS-9.4.TS1M5/product_data/base__94110_cd563__wx6__pt__sp0__1/pt_nls.xml",
         "Result" => "Operation Success",
      "EventName" => "Open Objet",
    "UserIsLocal" => "0",
       "HandleID" => "15397808166",

Also You may want to consider whether want to set store_xml to false otherwise it creates a lot of redundant data.

       store_xml => false
3

It works !

Thanks for the clarification.

With theses modifications the filter and date works as intended.

filter {
    xml {
       source => "message"
       force_array => false
       xpath => [
                 "/Event/System/EventID/text()", "EventID",
                 "/Event/System/EventName/text()", "EventName",
                 "/Event/System/Source/text()", "Source",
                 "/Event/System/Result/text()", "Result",
                 "/Event/System/TimeCreated/@SystemTime", "SystemTime",
                 "/Event/System/Computer/text()", "Computer",
                 "/Event/EventData/ClientIP/text()", "ClientIP",
                 "/Event/EventData/UnixID/text()", "UnixID",
                 "/Event/EventData/WindowsSID/text()", "WindowsSID",
                 "/Event/EventData/UserIsLocal/text()", "UserIsLocal",
                 "/Event/EventData/DomainName/text()", "DomainName",
                 "/Event/EventData/UserName/text()", "UserName",
                 "/Event/EventData/ObjectType/text()", "ObjectType",
                 "/Event/EventData/HandleID/text()", "HandleID",
                 "/Event/EventData/ObjectName/text()", "ObjectName",
                 "/Event/EventData/ReadOffSet/text()", "ReadOffset",
                 "/Event/EventData/ReadCount/text()", "ReadCount",
                 "/Event/EventData/UserACE/text()", "UserACE",
                 "/Event/EventData/UserACEMask/text()", "UserACEMask",
                 "/Event/EventData/DesiredAcces/text()", "DesiredAcces",
                 "/Event/EventData/Attributes/text()", "Attributes",
                 "/Event/EventData/WriteOffSet/text()", "WriteOffset",
                 "/Event/EventData/WriteCount/text()", "WriteCount",
                 "/Event/EventData/InformationSet/text()", "InformationSet",
                 "/Event/EventData/NewDirHandle/text()", "NewDirHandle",
                 "/Event/EventData/NewPath/text()", "NewPath",
                 "/Event/EventData/ACECount/text()", "ACECount",
                 "/Event/EventData/FailureReason/text()", "FailureReason"
       ]
       store_xml => false
       target => "doc"
    }

    date {
      match => ["SystemTime", "yyyy-MM-dd HH:mm:ss"]
          timezone => "Europe/Brussels"
          target => "@timestamp"
    }
}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.