# Hi, I am new to ELK. Trying to parse the date inside the multiline messages as timestamp. Let me know what config file to be used

**URL:** <https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792>\
**Category:** Logstash\
**Created:** [March 8, 2016, 3:08pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792 "2016-03-08T15:08:45Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nkpaul](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@nkpaul](https://discuss.elastic.co/u/nkpaul)\
**Post date:** [March 8, 2016, 3:08pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/1 "2016-03-08T15:08:45Z")

</div>

Message file will look like:

I. 2016/03/08 09:22:34. Hi My name is Nil.  
This is a test message  
I. 2016/03/08 09:22:35. This is the message 2  
I. 2016/03/08 09:22:36. This is multiline message 3  
message 3  
I. 2016/03/08 09:22:37. Thanks for all your help.

---

<div class="post-metadata">

**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\
**Post date:** [March 8, 2016, 4:53pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/2 "2016-03-08T16:53:21Z")

</div>

For that pattern try this:

```
input {
   file {
       path => "/path/to/log.file"
       type => logtest
       sincedb_path => "/dev/null"
       codec => multiline {
       pattern => "^I"
       negate => true
       what => previous
      }
}

filter {
   if [type] =~ "logtest" {
     grok {
       match => ["message", "I. %{SPACE}(?<runtime>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND})"]
     }
      
     date {
        match => ["runtime", "YYYY/MM/dd HH:mm:ss"]
        target => "@timestamp"
        }
     }
}

output {
    stdout { codec => rubydebug }

}

```

I am not sure if that is exactly what you need, but it will append any line that does not begin with "I." to the previous line, then create the timestamp from the date in the log entry.

---

<div class="post-metadata">

**Author:** ![nkpaul](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@nkpaul](https://discuss.elastic.co/u/nkpaul)\
**Post date:** [March 8, 2016, 6:47pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/3 "2016-03-08T18:47:13Z")

</div>

Hi Mick,  
Thanks for quick response. I have tried seems not working as desired.  
Given below is the conf file I am using also have given data sample. Always the line will start with either I. or E. or W.  
Thanks  
Nil

input {  
file {  
path =\> "/u01/REPLOG/_\_RS_.log\*"  
type =\> "REPLOG"  
#sincedb\_path =\> "/dev/null"  
codec =\> multiline {  
pattern =\> "^T"  
negate =\> true  
what =\> previous  
}  
}  
}

filter {  
if [type] =~ "REPLOG" {  
grok {  
match =\> { "path" =\> "%{GREEDYDATA}/%{GREEDYDATA:srvname}.log\*" }  
add\_field =\> { "RepSrvName" =\> "%{srvname}" }  
}

```
grok {
    match => ["message", "I. %{SPACE}(?<mytimestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:${SECOND})"]
}
date {
    match => ["mytimestamp", "YYYY/MM/dd HH:mm:ss"]
    target => "@timestamp"
}

```

}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
}  
stdout { codec =\> rubydebug }  
}

I. 2016/03/05 06:48:08. NIL TEST'  
Copyright 2014  
SAP AG or an SAP affiliate company. All rights reserved.  
Unpublished rights reserved under U.S. copyright laws.

This software contains confidential and trade secret information of SAP AG or  
an SAP affiliate company. Use, duplication or disclosure of the software and  
documentation by the U.S. Government is subject to restrictions set forth  
I. 2016/03/05 06:48:08. Using interfaces file '/irs\_rs/database/NIL\_RS/etc/interfaces'.  
I. 2016/03/05 06:48:08. Server name is 'NIL\_RS'.  
I. 2016/03/05 06:48:08. Replication Server memory model: 64bit.  
I. 2016/03/06 03:42:33. A grouped transaction of 2 individual transactions has failed in database 'IQSRV.GPST02'. It will be broken into smaller groups and retried.  
E. 2016/03/06 03:42:33. ERROR #1027 DSI EXEC(259(1) IQSRV.GPST01) - /generic/useful/cm.c(6157)  
Open Client Client-Library error: Error: 84083972, Severity 5 -- 'ct\_connect(): network packet layer: internal net library error: Net-Lib protocol driver call to connect two endpoints failed', Operating System error 146 -- 'Socket connect failed - errno 146 Connection refused'.  
E. 2016/03/06 03:42:33. ERROR #13045 DSI EXEC(259(1) IQSRV.GPST01) - /generic/useful/cm.c(6162)  
Failed to connect to server 'IQSRV' as user 'GPST01'. See CT-Lib and/or server error messages for more information.  
I. 2016/03/06 03:42:33. Trying to connect to server 'IQSRV' as user 'GPST01' ......  
E. 2016/03/06 03:42:33. ERROR #1027 DSI EXEC(260(1) IQSRV.GPST02) - /generic/useful/cm.c(6157)  
Open Client Client-Library error: Error: 84083972, Severity 5 -- 'ct\_connect(): network packet layer: internal net library error: Net-Lib protocol driver call to connect two endpoints failed', Operating System error 146 -- 'Socket connect failed - errno 146 Connection refused'.  
E. 2016/03/06 03:42:33. ERROR #13045 DSI EXEC(260(1) IQSRV.GPST02) - /generic/useful/cm.c(6162)  
Failed to connect to server 'IQSRV' as user 'GPST02'. See CT-Lib and/or server error messages for more information.  
I. 2016/03/06 03:42:33. Trying to connect to server 'IQSRV' as user 'GPST02' ......  
I. 2016/03/06 03:44:41. Still trying to connect to server 'IQSRV' as user 'GPST02' ......  
I. 2016/03/06 17:31:05. ...... connected to server 'RPTSRV02' as user 'GPS\_maint'.  
W. 2016/03/06 20:05:55. WARNING #5091 DSI EXEC(328(1) IQSRV.CLIENT\_REF) - neric/dsi/dsiqmint.c(4944)  
A transaction for database 'IQSRV.CLIENT\_REF' failed. It will be retried 5 times. The data server error received (#-157) is mapped to RETRY\_LOG or RETRY\_STOP.  
I. 2016/03/06 20:30:04. Connection to server 'IQSRV' as user 'CLIENT\_REF' has been faded out (closed).  
I. 2016/03/07 00:53:41. ...... connected to server 'IQSRV' as user 'CLIENT\_REF'.  
E. 2016/03/07 03:15:31. ERROR #1027 DSI EXEC(295(2) DW\_PRD.DW\_GPS00) - eneric/dsi/dsiutil.c(439)  
Open Client Client-Library error: Error: 84083975, Severity 5 -- 'ct\_send(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect'.  
E. 2016/03/07 03:15:31. ERROR #5215 DSI EXEC(295(2) DW\_PRD.DW\_GPS00) - eneric/dsi/dsiutil.c(451)  
The interface function 'RCIExecute' returns FAIL for database 'DW\_PRD.DW\_GPS00'. The errors are retryable. The DSI thread will restart automatically. See messages from the interface function for more information.  
I. 2016/03/07 03:24:43. Connection to server 'IQSRV' as user 'CLIENT\_REF' has been faded out (closed).  
I. 2016/03/07 03:31:09. ...... connected to server 'IQSRV' as user 'CLIENT\_REF'.

---

<div class="post-metadata">

**Author:** ![nkpaul](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@nkpaul](https://discuss.elastic.co/u/nkpaul)\
**Post date:** [March 9, 2016, 2:48pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/4 "2016-03-09T14:48:08Z")

</div>

Found the issue in my config file. Still it is giving parsing error.

{  
"@timestamp" =\> "2016-03-09T14:46:00.231Z",  
"message" =\> "I. 2016/02/05 15:06:06. Connection to server 'IFSDW\_PRD04' as user 'GPS00\_maint' has been faded out (closed). ",  
"@version" =\> "1",  
"path" =\> "/u01/REPLOG/C\_RS.log",  
"host" =\> "[itsrhv22112.it.statestr.com](http://itsrhv22112.it.statestr.com)",  
"type" =\> "REPLOG",  
"srvname" =\> "C\_RS",  
"RepSrvName" =\> "C\_RS",  
"tags" =\> [  
[0] "\_grokparsefailure"  
]  
}

CONFIG FILE:  
input {  
file {  
path =\> "/u01/REPLOG/_\_RS_.log\*"  
type =\> "REPLOG"  
#sincedb\_path =\> "/dev/null"  
codec =\> multiline {  
#pattern =\> ["^I.", "^W.", "^E."]  
pattern =\> "^I."  
#pattern =\> "^W."  
#pattern =\> "^E."  
negate =\> true  
what =\> previous  
}  
}  
}

filter {  
if [type] =~ "REPLOG" {  
grok {  
match =\> { "path" =\> "%{GREEDYDATA}/%{GREEDYDATA:srvname}.log\*" }  
add\_field =\> { "RepSrvName" =\> "%{srvname}" }  
}

```
grok {
    match => ["message", "I. %{SPACE}(?<mytimestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:${SECOND})"]
    #match => ["message", "W. %{SPACE}(?<mytimestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:${SECOND})"]
    #match => ["message", "E. %{SPACE}(?<mytimestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:${SECOND})"]
}
date {
    match => ['mytimestamp', 'YYYY/MM/dd HH:mm:ss']
    target => '@timestamp'
}

```

}  
}

output {  
elasticsearch {  
hosts =\> ["itsrhv22112:9200"]  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 9, 2016, 8:36pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/5 "2016-03-09T20:36:56Z")

</div>

> ```
> match => ["message", "I. %{SPACE}(?<mytimestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:${SECOND})"]
> 
> ```

Be systematic. Start with the simplest possible expression, e.g.

```
^I\.

```

and build from there until it stops working.

In this case I believe you'll notice that things stop working after you add %{SPACE} because in the example log message there's only one space after "I." but the expression above requires two.

---

<div class="post-metadata">

**Author:** ![nkpaul](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@nkpaul](https://discuss.elastic.co/u/nkpaul)\
**Post date:** [March 9, 2016, 9:11pm UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/6 "2016-03-09T21:11:51Z")

</div>

Thanks! It was my mistake. Somehow I put $ instead of % and it failed.

match =\> ["message", "I. (?%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND})"]

---

<div class="post-metadata">

**Author:** ![nkpaul](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@nkpaul](https://discuss.elastic.co/u/nkpaul)\
**Post date:** [March 15, 2016, 12:41am UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/7 "2016-03-15T00:41:43Z")

</div>

Unable to create index in ES from logstash

output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> ["127.0.0.1:9200"]  
index =\> "MyLog-%{+YYYY.MM.dd}"  
}  
}

Please let me know what am I missing.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2016, 6:33am UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/8 "2016-03-15T06:33:20Z")

</div>

Check the Logstash logs. Starting Logstash with `--verbose` might give additional clues.

Are you sure Logstash is reading anything? Previously I see that you have `sincedb_path => "/dev/null"` commented out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:07am UTC](https://discuss.elastic.co/t/hi-i-am-new-to-elk-trying-to-parse-the-date-inside-the-multiline-messages-as-timestamp-let-me-know-what-config-file-to-be-used/43792/9 "2017-07-06T05:07:01Z")

</div>