Unable to set parse and set timestamp with logstash (7.6.2) and xml filter plugin

Elastic Stack Logstash
1

Environment details: ELK stack 7.6.2 Windows 10

I am unable to replace/set an elastic search timestamp from my logs while indexing via Logstash. It simply adds as a new field and does not replace the original field. It simply adds "_dateparsefailure" tag without any other information.

I suspect the date filter is not working.

My sample log data:

<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819710045" level="WARN" thread="27"><log4j:message>registrarCheck.bookingWizardController.TryUpdatebookingCareOptions(): bookingCareOptionId: CenterBasedCare, bookingId: 5745493, bookingregistrarsCount: 5, IsEditbooking: False, IsEditbookingStep2Modified: False, IsMemberShip: False</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="283" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760731" level="ERROR" thread="15"><log4j:message>ERROR from EasyDraft API for funding accountid-&gt;0-&gt;Name: firstname lastname-&gt;Card number is invalid</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="139" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760856" level="ERROR" thread="15"><log4j:message>Error in controller: effective username: user1, identity username: user1, machine name: webserver1 
Client Name: [zzz Test ESomeApplication], Contract Id: [7ee17d62-d292-e511-b173-005056991898]
, Person Id: [143658262]
, Client ID: [b33442b3-d192-e511-b173-005056991898], Contract Relationship ID: [4529625]
, Person Fullname: [firstname lastname].
, Full Name: [firstname lastname], CRM ID: [a64c97b1-8a80-e811-b738-005056991899]</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.
   at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\SomeApplication\SomeOrganization.SomeApplication.BusinessLogic\PaymentAccount.cs:line 415
   at Csla.BusinessBase1.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 163
   at Csla.BusinessBase1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 350
   at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action1 updateModel, Boolean forceUpdate) in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.Web.ApplicationBlocks\Controllers\CustomCslaMvcController.cs:line 171</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530824089499" level="ERROR" thread="41"><log4j:message>Error Occured while Save Login in Class Login &amp; Method : Save For Username : tegh14</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-2-131752976869399121" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>System.Security.Authentication.AuthenticationException: We can�t find that username and/or password.  If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.
   at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.BusinessLogic\Security\Login.cs:line 547</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" timestamp="1587880949425" level="WARN" thread="47"><log4j:message>User mphilpunla-&gt;LoginWithSAML-&gt;lobuniqueId 19153694</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="webserver2" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132323544167926323" /><log4j:data name="log4net:UserName" value="SomeOrganisation\!svc-lob-apps" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="webserver2" /></log4j:properties><log4j:locationInfo class="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="E:\TFS2018agent\agent\_work\96\s\Shared\SomeOrganisation.Shared.ApplicationBlocks\Logging\Logging.cs" line="294" /></log4j:event>

My logstash configuration file:

input { 
    file {
      path => ["C:/Users/maskedUsername/Desktop/stackoverflow-log4net.txt"]
      start_position => "beginning"
      file_sort_by => "last_modified"
      file_sort_direction => "desc"
      sincedb_path => "NUL"
      type => "appl"
      codec => multiline {
          pattern => "^<log4j:event"
          negate => true
          what => "previous"
      }
    }
 }

filter {
  if [type] == "appl" {
    grok {
        add_tag => [ "groked" ]
        match => ["message", ".*"]
        remove_tag => ["_grokparsefailure"]
    }
    xml {
      source => "message"
      remove_namespaces => true
      target => "log4jevent"
      xpath => [ "//event/@timestamp", "timestamp" ]
      xpath => [ "//event/@level", "loglevel" ]
      xpath => [ "/event/message/text()", "message" ]
      xpath => [ "/event/throwable/text()", "exception" ]
      xpath => [ "//event/properties/data[@name='log4jmachinename']/@value", "machinename" ]
      xpath => [ "//event/properties/data[@name='log4japp']/@value", "app" ]
      xpath => [ "//event/properties/data[@name='log4net:UserName']/@value", "username" ]
      xpath => [ "//event/properties/data[@name='log4net:Identity']/@value", "identity" ]
      xpath => [ "//event/properties/data[@name='log4net:HostName']/@value", "hostname" ]
    }
    mutate {
      remove_field => ["type", "tags", "message"]
    }
    date {
        match => [ "timestamp","UNIX" ]
        target => "@timestamp"
        remove_field => ["timestamp"]
    }
  }
}

output {
  elasticsearch { 
  hosts => ["localhost:9200"] 
  index => "log4jevents"
  document_type => "log4jevent"
  }
  stdout { codec => rubydebug }
}
2

That is UNIX_MS, not UNIX.

What are you trying to achieve using the grok filter? It will unconditionally match and not create any fields.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.