Problems with config date {} multiline files

Elastic Stack Logstash
1

Hello everyone!
Any log file of the application. The application creates a single transaction = one log file. I use Logstash analysis of this file. The application repeatedly inserts timestamp in the log files. I make a minimal configuration file

@timestamp value gets the current time (at the time of run parsing) instead of the log file.
log time = 2014-12-30 15:03:48.025
@timestamp => "2015-05-26T07:47:50.618Z",
What to do in such a situation?
Acceptable workaround. How to take the first value "mytimestamp", other values can be ignored

input 
{stdin {}}

filter 
{
  grok {
    match => [ "message","(?<mytimestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})"]
  }
  date {
    match => [ "mytimestamp", "YYYY-MM-dd HH:mm:ss.SSS" ]
    locale => "en"
    add_tag => [ "tsmatch" ]
  }
 	multiline 
	{ pattern => "duration="
	negate => true
	what => "previous"}
}

output { stdout { codec => rubydebug }     }

debug output

{
        "message" => "\r\n2014-12-30 15:03:48.025\r\nUsername=bancadee_g2, requestTypeName=CreatePersonRequestMessage\r\n\r\n\r\n2014-12-30 15:03:48.212\r\nInvoking... invocationID=i15:03:48.0093s1, connection.U1Lo    gin=bancadee_g2, actualCert=8F2FADF310C72A9D37725C82A792, AppId=DeEconomiApplication\r\n          <d4p1:Code>UnknownError</d4p1:Code>\r\n          <d4p1:ID>2014-12-30 17:51:13.446s1</d4p1:ID>\r\n          <d4p1:Mes    sage>(Can't map transfer1 id=37598427, sTransferCode=28978489, msg=transfer id=38427 does not contain Amount with type=TransferAmountFixedInRC)</d4p1:Message>\r\n\r\n\r\n  <s:Header>\r\n\t<a:Username>bancadee_g2</a:Username>\r\n        </KeyInfo>\r\n      </e:EncryptedKey>\r",
       "@version" => "1",
     "@timestamp" => "2015-05-26T07:47:50.618Z",
           "host" => "SR-ELK01-S01-01",
           "tags" => [
        [0] "_grokparsefailure",
        [1] "multiline",
        [2] "tsmatch"
    ],
    "mytimestamp" => [
        [0] "2014-12-30 15:03:48.025",
        [1] "2014-12-30 15:03:48.212",
        [2] "2014-12-30 17:51:13.446"
    ]
}
{
       "message" => "duration=0,515625 seconds.\r",
      "@version" => "1",
    "@timestamp" => "2015-05-26T07:47:50.634Z",
          "host" => "SR-ELK01-S01-01",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

Part of the log with a few dates:

2014-12-30 15:03:48.025
Username=bancadee_g2, requestTypeName=CreatePersonRequestMessage


2014-12-30 15:03:48.212
Invoking... invocationID=i15:03:48.0093s1, connection.U1Login=bancadee_g2, actualCert=8F2FADF310C72A9D37725C82A792, AppId=DeEconomiApplication
          <d4p1:Code>UnknownError</d4p1:Code>
          <d4p1:ID>2014-12-30 17:51:13.446s1</d4p1:ID>
          <d4p1:Message>(Can't map transfer1 id=37598427, sTransferCode=28978489, msg=transfer id=38427 does not contain Amount with type=TransferAmountFixedInRC)</d4p1:Message>


  <s:Header>
	<a:Username>bancadeeconomi_g2</a:Username>
        </KeyInfo>
      </e:EncryptedKey>
duration=0,515625 seconds.
2

I have successfully solved this problem. You must first put a multiline processing, and then look for a date. In this case, the first occurrence is taken as @timestamp.

The final configuration

input 
{ stdin {} }
filter 
{
multiline 
	{
	pattern => "duration="
	negate => true
	what => "previous"
    	}
  grok {
    match => [ "message","(?<mytimestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})"]
  }
  date {
    match => [ "mytimestamp", "YYYY-MM-dd HH:mm:ss.SSS" ]
    locale => "en"
    remove_field => [ "mytimestamp" ]
  }
}
output { stdout { codec => rubydebug } }
1 Like