Unable to set parse and set timestamp with logstash (7.6.2) and xml filter plugin

Environment details: ELK stack 7.6.2 Windows 10

I am unable to replace/set an elastic search timestamp from my logs while indexing via Logstash. It simply adds as a new field and does not replace the original field. It simply adds "_dateparsefailure" tag without any other information.

I suspect the date filter is not working.

My sample log data:

<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819710045" level="WARN" thread="27"><log4j:message>registrarCheck.bookingWizardController.TryUpdatebookingCareOptions(): bookingCareOptionId: CenterBasedCare, bookingId: 5745493, bookingregistrarsCount: 5, IsEditbooking: False, IsEditbookingStep2Modified: False, IsMemberShip: False</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="283" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760731" level="ERROR" thread="15"><log4j:message>ERROR from EasyDraft API for funding accountid-&gt;0-&gt;Name: firstname lastname-&gt;Card number is invalid</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="139" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760856" level="ERROR" thread="15"><log4j:message>Error in controller: effective username: user1, identity username: user1, machine name: webserver1 
Client Name: [zzz Test ESomeApplication], Contract Id: [7ee17d62-d292-e511-b173-005056991898]
, Person Id: [143658262]
, Client ID: [b33442b3-d192-e511-b173-005056991898], Contract Relationship ID: [4529625]
, Person Fullname: [firstname lastname].
, Full Name: [firstname lastname], CRM ID: [a64c97b1-8a80-e811-b738-005056991899]</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.
   at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\SomeApplication\SomeOrganization.SomeApplication.BusinessLogic\PaymentAccount.cs:line 415
   at Csla.BusinessBase1.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 163
   at Csla.BusinessBase1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 350
   at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action1 updateModel, Boolean forceUpdate) in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.Web.ApplicationBlocks\Controllers\CustomCslaMvcController.cs:line 171</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530824089499" level="ERROR" thread="41"><log4j:message>Error Occured while Save Login in Class Login &amp; Method : Save For Username : tegh14</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-2-131752976869399121" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>System.Security.Authentication.AuthenticationException: We can�t find that username and/or password.  If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.
   at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.BusinessLogic\Security\Login.cs:line 547</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" timestamp="1587880949425" level="WARN" thread="47"><log4j:message>User mphilpunla-&gt;LoginWithSAML-&gt;lobuniqueId 19153694</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="webserver2" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132323544167926323" /><log4j:data name="log4net:UserName" value="SomeOrganisation\!svc-lob-apps" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="webserver2" /></log4j:properties><log4j:locationInfo class="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="E:\TFS2018agent\agent\_work\96\s\Shared\SomeOrganisation.Shared.ApplicationBlocks\Logging\Logging.cs" line="294" /></log4j:event>

My logstash configuration file:

input { 
    file {
      path => ["C:/Users/maskedUsername/Desktop/stackoverflow-log4net.txt"]
      start_position => "beginning"
      file_sort_by => "last_modified"
      file_sort_direction => "desc"
      sincedb_path => "NUL"
      type => "appl"
      codec => multiline {
          pattern => "^<log4j:event"
          negate => true
          what => "previous"
      }
    }
 }

filter {
  if [type] == "appl" {
    grok {
        add_tag => [ "groked" ]
        match => ["message", ".*"]
        remove_tag => ["_grokparsefailure"]
    }
    xml {
      source => "message"
      remove_namespaces => true
      target => "log4jevent"
      xpath => [ "//event/@timestamp", "timestamp" ]
      xpath => [ "//event/@level", "loglevel" ]
      xpath => [ "/event/message/text()", "message" ]
      xpath => [ "/event/throwable/text()", "exception" ]
      xpath => [ "//event/properties/data[@name='log4jmachinename']/@value", "machinename" ]
      xpath => [ "//event/properties/data[@name='log4japp']/@value", "app" ]
      xpath => [ "//event/properties/data[@name='log4net:UserName']/@value", "username" ]
      xpath => [ "//event/properties/data[@name='log4net:Identity']/@value", "identity" ]
      xpath => [ "//event/properties/data[@name='log4net:HostName']/@value", "hostname" ]
    }
    mutate {
      remove_field => ["type", "tags", "message"]
    }
    date {
        match => [ "timestamp","UNIX" ]
        target => "@timestamp"
        remove_field => ["timestamp"]
    }
  }
}

output {
  elasticsearch { 
  hosts => ["localhost:9200"] 
  index => "log4jevents"
  document_type => "log4jevent"
  }
  stdout { codec => rubydebug }
}

That is UNIX_MS, not UNIX.

What are you trying to achieve using the grok filter? It will unconditionally match and not create any fields.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.