"6508 12:31:48 INFO Heartbeat - Worker thread started", "@version"=>"1", "@timestamp"=>"2017-04-10T07:01:48.000Z", "beat"=>{"version"=>"5.3.0", "name"=>"delvmplssmdo3.sapient.com", "hostname"=>"delvmplssmdo3.sapient.com"}, "source"=>"/app/logs/sitecore/log.20170410.123146.txt", "offset"=>54, "type"=>"sitecore_log", "input_type"=>"log", "host"=>"delvmplssmdo3.sapient.com", "tags"=>["beats_input_codec_plain_applied"], "pid"=>"6508", "time"=>"12:31:48", "loglevel"=>"INFO", "sitecore_log"=>" Heartbeat - Worker thread started", "timestamp"=>"2017/04/10 12:31:48"}, @lut={"@timestamp"=>[{"message"=>"6508 12:31:48 INFO Heartbeat - Worker thread started", "@version"=>"1", "@timestamp"=>"2017-04-10T07:01:48.000Z", "beat"=>{"version"=>"5.3.0", "name"=>"delvmplssmdo3.sapient.com", "hostname"=>"delvmplssmdo3.sapient.com"}, "source"=>"/app/logs/sitecore/log.20170410.123146.txt", "offset"=>54, "type"=>"sitecore_log", "input_type"=>"log", "host"=>"delvmplssmdo3.sapient.com", "tags"=>["beats_input_codec_plain_applied"], "pid"=>"6508", "time"=>"12:31:48", "loglevel"=>"INFO", "sitecore_log"=>" Heartbeat - Worker thread started", "timestamp"=>"2017/04/10 12:31:48"}, "@timestamp"]}>>], :response=>{"index"=>{"_index"=>"filebeat-2017.04.10", "_type"=>"sitecore_log", "_id"=>"AVuK8QzJwNDRIOvARbFQ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017/04/10 12:31:48\" is malformed at \"/04/10 12:31:48\""}}}}, :level=>:warn, :file=>"logstash/outputs/elasticsearch/common.rb", :line=>"119", :method=>"submit"}`
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [April 20, 2017, 10:59am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/8 "2017-04-20T10:59:35Z")
Hi Magnus,
Thanks for all your help.
I am able to parse the date and concatenate timestamp from logs now.
Cheers 🙂
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 11, 2017, 12:43pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/9 "2017-05-11T12:43:26Z")
Hi Magnus,
I am facing issue when parsing multiple files to fetch timestamp from filename.
My filename is in this format:
MyCustomLogFileAppender\_07.05.2017.log
MyCustomLogFileAppender\_08.05.2017.log
MyCustomLogFileAppender\_08.05.2017.log.1
Filebeat config:
> ```
> -
> paths:
> - /app/logs/sitecore/MyCustomLogFileAppender_*
> input_type: log
> document_type: sitecore_log
> multiline:
> pattern: '^[[:space:]]+|^[[:space:]]+at+|^Exception:|^Source:|^Nested+|^Message:|^D:'
> negate: false
> match: after
>
> ```
Below is my logstash config:
> if [type] == "sitecore\_log" {
> multiline {
> patterns\_dir =\> "/etc/logstash/conf.d/patterns"
> pattern =\> "^%{INT:pid} %{TIME:time}"
> negate =\> true
> what =\> previous
> max\_age =\> 10
> }
> grok {
> patterns\_dir =\> "/etc/logstash/conf.d/patterns"
> match =\> ["message", "(%{INT:pid}|%{WORD:heartbeat}) %{TIME:time} %{LOGLEVEL:loglevel} (%{GREEDY\_DATA:sitecore\_log}|%{GREEDY\_DATA:rewrite\_log})"]
> }
> grok {
> match =\> ["source", "(\_%{MONTHDAY:day}.%{MONTHNUM:month}.%{YEAR:year})"]
> add\_field =\> ["timestamp", "%{day}-%{month}-%{year} %{time}"]
> }
> mutate {
> remove\_field =\> ["day", "month", "year"]
> }
> date {
> match =\> ["timestamp", "dd-MM-yyy HH:mm:ss.SSSZ", "dd-MM-yyyy HH:mm:ss"]
> target =\> "@timestamp"
> }
I am getting multiple timestamp in kibana:
> {
> "\_index": "filebeat-2017.05.11",
> "\_type": "sitecore\_log",
> "\_id": "AVv3hNwFqdh28cPIV0fK",
> "\_score": null,
> "\_source": {
> "message": "3512 00:04:24 DEBUG [BucketLinkManager] - Method started :5/9/2017 12:04:24 AM\n 924 09:46:09 DEBUG BucketItemResolver Start",
> "@version": "1",
> "@timestamp": "2017-05-11T12:38:21.131Z",
> "offset": [
> 2645811,
> 2256146
> ],
> "type": "sitecore\_log",
> "input\_type": "log",
> "beat": {
> "name": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "hostname": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "version": "5.3.0"
> },
> "source": [
> "/app/logs/sitecore/MyCustomLogFileAppender\_09.05.2017.log.3",
> "/app/logs/sitecore/MyCustomLogFileAppender\_08.05.2017.log.4"
> ],
> "host": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "tags": [
> "beats\_input\_codec\_plain\_applied",
> "multiline",
> "\_dateparsefailure"
> ],
> "pid": "3512",
> "time": "00:04:24",
> "loglevel": "DEBUG",
> "sitecore\_log": "[BucketLinkManager] - Method started :5/9/2017 12:04:24 AM\n 924 09:46:09 DEBUG BucketItemResolver Start",
> "timestamp": "09,08-05,05-2017,2017 00:04:24"
> },
> "fields": {
> "@timestamp": [
> 1494506301131
> ]
> },
> "highlight": {
> "type.raw": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ],
> "type": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1494506301131
> ]
> }
Can you please suggest something to resolve this. I am using multiline filter to parse the multiline stacktrace.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 11, 2017, 12:52pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/10 "2017-05-11T12:52:46Z")
Why are you using multiline both in Filebeat and Logstash? What do you mean with "multiple timestamps"?
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 11, 2017, 1:02pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/11 "2017-05-11T13:02:25Z")
Hi Magnus,
My log event does not contains datestamp, datestamp is being generated in the filename.(MyCustomLogFileAppender\_07.05.2017.log). My log event looks like this:
> 3512 00:04:24 DEBUG [BucketLinkManager] - Method started :5/9/2017 12:04:24 AM
So i want the datestamp to be taken from filename and time from event log to generate @timestamp.
Its working for a single log file, but when i am trying to parse multiple log files the timestamp being generated like:
> "timestamp": "09,08-05,05-2017,2017 00:04:24"
I am using multiline in filebeat and logstash to parse the multiline stack trace. Please suggest if that could be done by using multiline in single logstash configuration.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 11, 2017, 1:46pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/12 "2017-05-11T13:46:20Z")
> I am using multiline in filebeat and logstash to parse the multiline stack trace. Please suggest if that could be done by using multiline in single logstash configuration.
Do it on the Filebeat side. I don't understand why you'd need to have multiline processing in two places.
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 12, 2017, 5:36pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/13 "2017-05-12T17:36:42Z")
Hi Magnus,
Greetings for the Day!!
As suggested, i have removed the multiline config from logstash and placed it on filebeat.:
> ```
> paths:
> - /app/logs/sitecore/MyCustomLogFileAppender_*
> input_type: log
> document_type: sitecore_log
> multiline:
> pattern: '^[[:digit:]]{4}[[:space:]][[:digit:]]{2}\:[[:digit:]]{2}\:[[:digit:]]{2}'
> negate: true
> match: previous
>
> ```
Logstash config:
> if [type] == "sitecore\_log" {
> grok {
> patterns\_dir =\> "/etc/logstash/conf.d/patterns"
> match =\> ["message", "(%{INT:pid}|%{WORD:heartbeat}) %{TIME:time} %{LOGLEVEL:loglevel} (%{GREEDY\_DATA:sitecore\_log}|%{GREEDY\_DATA:rewrite\_log})"]
> }
> grok {
> match =\> ["source", "(\_%{MONTHDAY:day}.%{MONTHNUM:month}.%{YEAR:year})"]
> add\_field =\> ["timestamp", "%{day}-%{month}-%{year} %{time}"]
> }
> mutate {
> remove\_field =\> ["day", "month", "year"]
> }
> date {
> match =\> ["timestamp", "dd-MM-yyy HH:mm:ss.SSSZ", "dd-MM-yyyy HH:mm:ss"]
> target =\> "@timestamp"
> }
> }
Log Event:
> 3512 05:11:00 ERROR Application Error
> Exception: SolrNet.Exceptions.SolrConnectionException
> Message: \
> Source: SolrNet
> at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable`1 parameters) at SolrNet.Impl.SolrQueryExecuter`1.Execute(ISolrQuery q, QueryOptions options)
> at EuropeanTour.Repository.Implementation.SolrRepository`1.GetArticleFromItemId(String articleItemId, Int32 start, Int32 numberOfArticle, Int32& numFound) at EuropeanTour.Repository.Implementation.Global.ArticlesRepository.GetMediaList(String datasource, String variation, Int32 start, Int32 recordsToFetch, Boolean isAjax, Nullable`1 advertisementIndex, String currentItemID)
> at EuropeanTour.Controllers.Europeantour.ArticlesController.GetMediaList()
> Nested Exception
> Exception: System.Net.WebException
> Message: The remote server returned an error: (400) Bad Request.
> Source: System
> at System.Net.HttpWebRequest.GetResponse()
> at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
> at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)
> at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable`1 parameters)
Log Event:
> {
> "\_index": "filebeat-2017.05.12",
> "\_type": "sitecore\_log",
> "\_id": "AVv9sgd4qdh28cPIquG3",
> "\_score": null,
> "\_source": {
> "message": " at EuropeanTour.Repository.Implementation.TournamentRepository.GetTournamentStatus(Int32 tournamentId)",
> "@version": "1",
> "@timestamp": "2017-05-12T17:25:31.376Z",
> "source": "/app/logs/sitecore/MyCustomLogFileAppender\_08.05.2017.log.2",
> "offset": 61500,
> "type": "sitecore\_log",
> "input\_type": "log",
> "beat": {
> "name": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "hostname": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "version": "5.3.0"
> },
> "host": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "tags": [
> "beats\_input\_codec\_plain\_applied",
> "\_grokparsefailure",
> "\_dateparsefailure"
> ],
> "timestamp": "08-05-2017 %{time}"
> },
> "fields": {
> "@timestamp": [
> 1494609931376
> ]
> },
> "highlight": {
> "type.raw": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ],
> "type": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1494609931376
> ]
> }
I am trying to achieve two things here:
1. Need to generate @timestamp by concatenating date from filename and time from log event.
2. Parse multiline stacktrace in logstash.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 15, 2017, 5:23am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/14 "2017-05-15T05:23:08Z")
Your multiline configuration isn't working so let's start there. As documented, the `match` option should be efter "before" or "after". Not "previous".
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 15, 2017, 7:17am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/15 "2017-05-15T07:17:45Z")
Hi Magnus,
Thanks for your reply.
As suggested, i have modified the filebeat config and replaced 'previous' with 'before' and negate from 'true' to 'false'. Config now looks like:
> ```
> -
> paths:
> - /app/logs/sitecore/MyCustomLogFileAppender_*
> input_type: log
> document_type: sitecore_log
> multiline:
> pattern: '^[[:digit:]]{4}[[:space:]][[:digit:]]{2}\:[[:digit:]]{2}\:[[:digit:]]{2}'
> negate: false
> match: before
>
> ```
It seems my multiline filter is not working.
I have tried with below config as well using different pattern:
```
multiline:
pattern: '^[[:space:]]+|^[[:space:]]+at+|^Exception:|^Source:|^Nested+|^Message:|^D:'
negate: true
match: before
```
Still it does'nt seems to be working.
I am using filebeat version: filebeat version 5.3.0
Can you suggest if there is any syntax config changes for this particular version.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 15, 2017, 7:48am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/16 "2017-05-15T07:48:55Z")
The first thing you attempted is closer to what you need. Surely `negate` should be true, i.e. if the line _does not_ start with the timestamp it should be joined with the preceding line?
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 15, 2017, 8:47am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/17 "2017-05-15T08:47:20Z")
Hi Magnus,
Yes, in my case as you stated. negate should be true,
i.e. if the line does not start with the timestamp it should be joined with the preceding line (That's Correct)
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 15, 2017, 12:29pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/18 "2017-05-15T12:29:10Z")
Hi Magnus,
After making several attempts to make it work, I have this configuration that somehow worked.
```
-
paths:
- /app/logs/sitecore/MyCustomLogFileAppender*
input_type: log
document_type: sitecore_log
multiline.pattern: '[:digit:]]{3,4}[[:space:]][[:digit:]]{2}\:[[:digit:]]{2}\:[[:digit:]]{2}[[:space:]][[:alpha:]]{4,5}'
multiline.negate: true
multiline.match: after
```
Now, the issue is: in message field its taking multiple log events under single event:
I am pasting a small portion of log event, because the actual log event is very large:
> {
> "\_index": "filebeat-2017.05.06",
> "\_type": "sitecore\_log",
> "\_id": "AVwMC\_nrqdh28cPI1bVE",
> "\_score": null,
> "\_source": {
> "message": "2044 02:15:52 DEBUG CustomItemResolver Start\n2892 02:15:52 DEBUG CustomItemResolver Start\n3076 02:15:52 DEBUG CustomItemResolver Start\n3200 02:15:52 DEBUG CustomItemResolver Start\n3076 02:15:52 DEBUG LanguageItemResolver Start\n2044 02:15:52 DEBUG LanguageItemResolver Start\n2892 02:15:52 DEBUG LanguageItemResolver Start\n3076 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n2044 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n2892 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n3200 02:15:52 DEBUG LanguageItemResolver Start\n3200
> "@version": "1",
> "@timestamp": "2017-05-06T20:45:52.000Z",
> "source": "/app/logs/sitecore/MyCustomLogFileAppender\_07.05.2017.log",
> "offset": 31300055,
> "type": "sitecore\_log",
> "input\_type": "log",
> "beat": {
> "name": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "hostname": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "version": "5.3.0"
> },
> "host": "[delvmplssmdo3.sapient.com](http://delvmplssmdo3.sapient.com)",
> "tags": [
> "beats\_input\_codec\_plain\_applied"
> ],
> "pid": "2044",
> "time": "02:15:52",
> "loglevel": "DEBUG",
> "sitecore\_log": "CustomItemResolver Start\n2892 02:15:52 DEBUG CustomItemResolver Start\n3076 02:15:52 DEBUG CustomItemResolver Start\n3200 02:15:52 DEBUG CustomItemResolver Start\n3076 02:15:52 DEBUG LanguageItemResolver Start\n2044 02:15:52 DEBUG LanguageItemResolver Start\n2892 02:15:52 DEBUG LanguageItemResolver Start\n3076 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n2044 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n2892 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n3200 02:15:52 DEBUG LanguageItemResolver Start\n3200 02:15:52 DEBUG LanguageItemResolver Context Item is Empty and Language Code is not Correct\n3076 02:15:52 DEBUG BucketItemResolver Start\n
> "timestamp": "07-05-2017 02:15:52"
> },
> "fields": {
> "@timestamp": [
> 1494103552000
> ]
> },
> "highlight": {
> "type.raw": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ],
> "type": [
> "@kibana-highlighted-field@sitecore\_log@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1494103552000
> ]
> }
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 15, 2017, 12:35pm UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/19 "2017-05-15T12:35:33Z")
You need one more square bracket at the beginning of the expression (`[[:digit:]]`, not `[:digit:]]`). You'll also want to begin your regexp with `^`.
---
### Author:  [@shammi](https://discuss.elastic.co/u/shammi)
#### Post date: [May 16, 2017, 5:08am UTC](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556/20 "2017-05-16T05:08:13Z")
Hi Marcus,
Good Morning,
Yes, you are correct. I missed that square bracket in the beginning. I have made some tweaks in the config to make it working. Finally, the working config is:
```
paths:
- /app/logs/sitecore/MyCustomLogFileAppender*
input_type: log
document_type: sitecore_log
multiline.pattern: '[[:digit:]]{3,4}[[:space:]][[:digit:]]{2}\:[[:digit:]]{2}\:[[:digit:]]{2}[[:space:]][[:alpha:]]{4,5}'
multiline.negate: true
multiline.match: after
```
Also, in logstash logs i am getting below error:
> {:timestamp=\>"2017-05-16T10:34:26.667000+0530", :message=\>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: \5671, ssl=\>false, codec=\>\"UTF-8"\>, host=\>"0.0.0.0", ssl\_verify\_mode=\>"none", include\_codec\_tag=\>true, ssl\_handshake\_timeout=\>10000, congestion\_threshold=\>5, target\_field\_for\_codec=\>"message", tls\_min\_version=\>1, tls\_max\_version=\>1.2, cipher\_suites=\>["TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256"], client\_inactivity\_timeout=\>60\>\n Error: event executor terminated", :level=\>:error}
Kindly suggest if i can ignore it or anything needs to be taken care.
Versions i am using:
Filebeat: 5.3.0
Logstash: 2.4.1
[Next page](https://discuss.elastic.co/t/generate-timestamp-in-logstash-by-concatenating-date-from-filename-and-time-from-logs/82556.md?page=2)