# Timestamp replace not working

**URL:** <https://discuss.elastic.co/t/timestamp-replace-not-working/93350>\
**Category:** Logstash\
**Created:** [July 17, 2017, 7:40am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350 "2017-07-17T07:40:50Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 17, 2017, 7:40am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/1 "2017-07-17T07:40:51Z")

</div>

Hi, I understand this might be a very common question asked but I couldn't find any answer that addressed my issue.

Here it goes. I have a log message as shown below that I am publishing from Filebeats into Logstash.  
20170128 144622.584437 0005B9427CA0\_CU\_1 [user.info](http://user.info) sometext.

Problem: The timestamp used is still the current one and not corresponding to the log message.

I have defined a filter as below:  
filter {  
date {  
match =\> ["timestamp" , "yyyyMMdd HHmmss.SSS"]  
target =\> "@timestamp"  
add\_field =\> { "debug" =\> "timestampMatched"}  
}  
}

Logstash log messages:  
[2017-07-17T12:58:42,473][DEBUG][logstash.pipeline] filter received {"event"=\>{"@timestamp"=\>2017-07-17T07:28:41.042Z, "offset"=\>746730, "@version"=\>"1", "beat"=\>{"hostname"=\>"node1", "name"=\>"node1", "version"=\>"5.5.0"}, "input\_type"=\>"log", "host"=\>"node1", "source"=\>"/root/samplelogs/Debug.log", "message"=\>"20170128 144622.584437 0005B9427CA0\_CU\_1 [user.info](http://user.info) sometext", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied"]}}

[2017-07-17T12:58:42,473][DEBUG][logstash.pipeline] output received {"event"=\>{"@timestamp"=\>2017-07-17T07:28:41.042Z, "offset"=\>746730, "@version"=\>"1", "beat"=\>{"hostname"=\>"node1", "name"=\>"node1", "version"=\>"5.5.0"}, "input\_type"=\>"log", "host"=\>"node1", "source"=\>"/root/samplelogs/Debug.log", "message"=\>"20170128 144622.584437 0005B9427CA0\_CU\_1 [user.info](http://user.info) sometext", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied"]}}

What seems to be the issue here? Is the timestamp format missing something?

-Thanks  
Nikhil

---

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 17, 2017, 8:23am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/2 "2017-07-17T08:23:20Z")

</div>

Edit: Updated the post after removing incorrect ":" character. Issue is still not resolved.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 17, 2017, 9:31am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/4 "2017-07-17T09:31:12Z")

</div>

Where's the `timestamp` field that you're asking the date filter to parse? How are you creating that field?

---

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 17, 2017, 9:55am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/5 "2017-07-17T09:55:53Z")

</div>

> [@Nikhil\_Utane](#):
>
> 20170128 144622.584437

Its at the beginning of the line. It is created by syslogd. Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 17, 2017, 4:32pm UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/6 "2017-07-17T16:32:58Z")

</div>

Yes, I know it's in the `message` field but according to the log entry you posted,

> [2017-07-17T12:58:42,473][DEBUG][logstash.pipeline] output received {"event"=\>{"@timestamp"=\>2017-07-17T07:28:41.042Z, "offset"=\>746730, "@version"=\>"1", "beat"=\>{"hostname"=\>"node1", "name"=\>"node1", "version"=\>"5.5.0"}, "input\_type"=\>"log", "host"=\>"node1", "source"=\>"/root/samplelogs/Debug.log", "message"=\>"20170128 144622.584437 0005B9427CA0\_CU\_1 user.info sometext", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied"]}}

the event has no `timestamp` field which would explain why the date filter isn't working.

---

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 18, 2017, 3:38am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/7 "2017-07-18T03:38:08Z")

</div>

Ohh. I thought the filter will match if the timestamp is part of the message. So does it mean I have to first parse the message and extract the timestamp into a separate field and then apply the replace operation?  
In other words, have 2 filters, one grok to extract and then date to replace?  
Thanks.

---

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 18, 2017, 5:12am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/8 "2017-07-18T05:12:33Z")

</div>

Hi Magnus,

I modified my filter as below:

> ```
> filter {
> grok {
> patterns_dir => ["/root/logstash-5.5.0/patterns"]
> match => { "message" => "^%{CSSYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_hostname} %{DATA:syslog_level} %{DATA:app_name}: %{GREEDYDATA:syslog_message}" }
> }
> 
> ```

> ```
> date {
> match => ["syslog_timestamp" , "yyyyMMdd HHmmss.SSS"]
> target => "@timestamp"
> add_field => { "debug" => "timestampMatched"}
> }
> }
> 
> ```

```
> [2017-07-18T10:25:01,152][DEBUG][logstash.pipeline] filter received {"event"=>{"@timestamp"=>2017-07-18T04:54:55.170Z, "offset"=>747452, "@version"=>"1", "input_type"=>"log", "beat"=>{"hostname"=>"node1", "name"=>"node1", "version"=>"5.5.0"}, "host"=>"node1", "source"=>"/root/samplelogs/Debug.log", "message"=>"20170119 144002.184140 0005B9427CA0_CU_1 user.notice ProcMon:: 10.220.0.13 is valid IP Address", "type"=>"log", "tags"=>["beats_input_codec_plain_applied"]}}
> [2017-07-18T10:25:01,154][DEBUG][logstash.filters.grok] Running grok filter {:event=>2017-07-18T04:54:55.170Z node1 20170119 144002.184140 0005B9427CA0_CU_1 user.notice ProcMon:: 10.220.0.13 is valid IP Address}
> [2017-07-18T10:25:01,159][DEBUG][logstash.filters.grok] Event now: {:event=>2017-07-18T04:54:55.170Z node1 20170119 144002.184140 0005B9427CA0_CU_1 user.notice ProcMon:: 10.220.0.13 is valid IP Address}
> [2017-07-18T10:25:01,165][DEBUG][logstash.pipeline] output received {"event"=>{"MSEC"=>"184140", "offset"=>747452, "input_type"=>"log", "source"=>"/root/samplelogs/Debug.log", "message"=>"20170119 144002.184140 0005B9427CA0_CU_1 user.notice ProcMon:: 10.220.0.13 is valid IP Address", "type"=>"log", "syslog_message"=>"10.220.0.13 is valid IP Address", "tags"=>["beats_input_codec_plain_applied", "_dateparsefailure"], "app_name"=>"ProcMon:", "@timestamp"=>2017-07-18T04:54:55.170Z, "syslog_hostname"=>"0005B9427CA0_CU_1", "syslog_timestamp"=>"20170119 144002.184140", "@version"=>"1", "beat"=>{"hostname"=>"node1", "name"=>"node1", "version"=>"5.5.0"}, "host"=>"node1", "syslog_level"=>"user.notice"}}

```

My new timestamp field (syslog\_timestamp) is now created. However the @timestamp value is still not replaced. What am I doing wrong?  
Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 18, 2017, 6:21am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/9 "2017-07-18T06:21:55Z")

</div>

The problem might be that your timestamp contains microseconds but the "SSS" in your date pattern only capture milliseconds. Try using "SSSSSS" instead, or strip the last three digits with a mutate filter's gsub option. Elasticsearch's resolution is just milliseconds anyway.

---

<div class="post-metadata">

**Author:** ![Nikhil\_Utane](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nikhil_utane/32/27547_2.png) [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\
**Post date:** [July 18, 2017, 7:34am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/10 "2017-07-18T07:34:03Z")

</div>

Yes, I found the problem too and was going to reply the same here. 🙂

"Found the issue. I was only using upto 3 decimals (SSS) because of this text mentioned in the reference doc.  
"S: fraction of a second Maximum precision is milliseconds (SSS). Beyond that, zeroes are appended."  
But it turns out you still need to use 6 S's while defining the filter. "

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2017, 7:34am UTC](https://discuss.elastic.co/t/timestamp-replace-not-working/93350/11 "2017-08-15T07:34:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.