# How to convert existing timestamp format to another timestamp format using logstash?

**URL:** https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889
**Category:** Logstash
**Created:** [June 1, 2017, 11:13am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889 "2017-06-01T11:13:14Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![yash\_mangla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yash_mangla/32/25889_2.png) [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)
#### Post date: [June 1, 2017, 11:13am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/1 "2017-06-01T11:13:14Z")

</div>

Hi Guyz,

i have 2 different application logs.

one is containing (application1)

```
16/02/2017 19:21:19:452 INFO - 9DDF7A1C53D518BC9C4F7DAE9045C70E:/RCOM_PREPAID : ** Popped Stack Frame [/RCOM_PREPAID]
16/02/2017 19:21:19:467 INFO - C12A2AB5A5FF56B91AFDF8EF21107573:/RCOM_PREPAID : Storing :session___channel to complex: session:channel as []
16/02/2017 19:21:19:467 WARN - 1832809EE1D72F1CC4C83A2206D20FFA:/RCOM_PREPAID : - Particular Property 'rePromptNoInput' is not set!!!
16/02/2017 19:21:19:467 DEBUG - DDEAA825B49F320DE1CEF901032DEE2C:/RCOM_PREPAID : PlatformParams:getPlatformParams:Timestamp is 12/21/16 2:12:12
16/02/2017 19:21:19:467 ERROR - 75EC5FA713110FDC5B15A69A373257E6:/RCOM_PREPAID : session id:cgrmpp04-2017047134807-332 | Error processing request

```

and another application (application2) these logs:

```
2017-04-20 11:38:53,751 [Line171] INFO com.gl.nortel.ivr.service.invoker.cdb.CDBServiceHandlerMOB invokeSOAPRPCService- RequestMsg:<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><getCustomerProfile xmlns="http://webservices.ivr.ibm.com"><profileDTO xmlns=""><errordto xsi:nil="true"/><lob>prepaid</lob><clid>8173077459</clid><firstName xsi:nil="true"/><middleName xsi:nil="true"/><lastName xsi:nil="true"/><nameTitle xsi:nil="true"/><customerEmail xsi:nil="true"/><segmentdto xsi:nil="true"/><accountStatus xsi:nil="true"/><accountNo xsi:nil="true"/><doa xsi:nil="true"/><dob xsi:nil="true"/><language xsi:nil="true"/><tpin xsi:nil="true"/><region xsi:nil="true"/><incomingSource>IVR121</incomingSource><languageUpdated xsi:nil="true"/><tpinUpdated xsi:nil="true"/><dthProfileDTO xsi:nil="true"/><postpaidProfileDTO xsi:nil="true"/><prepaidProfileDTO xsi:nil="true"/><telemediaProfileDTO xsi:nil="true"/><source xsi:nil="true"/><circleId xsi:nil="true"/><airtelOne xsi:nil="true"/><residentialCity xsi:nil="true"/><correspondanceCity xsi:nil="true"/><MobilityServiceItemList xsi:nil="true"/></profileDTO></getCustomerProfile></soapenv:Body></soapenv:Envelope>
2017-04-20 11:38:53,752 [Line171] INFO com.gl.nortel.ivr.service.invoker.cdb.CDBServiceHandlerMOB invokeSOAPRPCService- ResponseMsg: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><getCustomerProfileResponse xmlns="http://webservices.ivr.ibm.com">

```

here, both timestamp is different and i want to convert both timestamp format to standard timestamp format  
which would be `"dd-mm-yyyy-HH-MM-SS"`

i am using different filebeat multiline setting for both applications  
application1:  
\> multiline.pattern: '\[1\]{2}/[0-9]{2}/[0-9]{4}'  
\> multiline.negate: true  
\> multiline.match: after

application2:

> multiline.pattern: '\[2\]{4}-[0-9]{2}-[0-9]{2}'  
> multiline.negate: true  
> multiline.match: after

and filter setting is

application 1 is:

```
grok {
	  match => { "message" => "%{DATESTAMP:timestamp} (?<logLevel>(?:DEBUG|FATAL|ERROR| WARN| INFO)) - (?<sessionId>(%{WORD})):\/(?<appName>(%{USERNAME})) : %{GREEDYDATA:errMsg}" }
	  
	}
	date {
		match => ["timestamp", "dd/mm/yyyy HH:mm:ss a", "dd/mm/yyyy HH:mm:ss:SSS"]
	}

```

and application 2 is:

```
grok {
	  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[(?<lineNum>(%{WORD}))\] (?<logLevel>(?:DEBUG|FATAL|ERROR|WARN|INFO)) %{SPACE}%{JAVACLASS:className} %{WORD:serviceMethod}- %{GREEDYDATA:errMsg}" }
	  
	}
	date {
		match => ["timestamp", "yyyy-mm-dd HH:mm:ss,SSS", "yyyy-mm-dd HH:mm:ss a"]
	}

```

you can test these pattern on [test pattern](http://grokconstructor.appspot.com/do/match#result)  
during test pattern please enable multiline negation true

please suggest some way which helps i can convert these logs timestamp format to another format  
like if format is  
`"yyyy-mm-dd HH:mm:ss,SSS" => "dd-mm-yyyy-HH-MM-SS"`

Good answer will be appreciated.

* * *

1. 0-9 

2. 0-9

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 1, 2017, 8:25pm UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/2 "2017-06-01T20:25:44Z")

</div>

There's no stock plugin for this. You'll have to write a snippet of Ruby code in a ruby filter.

---

<div class="post-metadata">

### Author: ![yash\_mangla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yash_mangla/32/25889_2.png) [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)
#### Post date: [June 2, 2017, 7:26am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/3 "2017-06-02T07:26:30Z")

</div>

alright @magnusbaeck

Is it possible if i break the existing timestamp to something like:  
suppose if we have these format  
`2017-04-20 11:38:53,751` then i can do something like in grok pattern  
`%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:min}:%{SECOND:sec},%{INT:usec}`

and then we add into another format something like

> ```
> mutate {
> add_field => {
> "timestamp" => "%{day}-%{month}-%{year}-%{hour}-%{min}-%{sec}"
> }
> remove_fields => ["day", "month", "year", "hour", "min", "sec", "usec"]
> }
> 
> ```

is it possible and if it possible  
then how to assign this field as a timestamp at same time ?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 7:32am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/4 "2017-06-02T07:32:36Z")

</div>

> is it possible and if it possible

Sure, you can do that.

> then how to assign this field as a timestamp at same time ?

You mean how to get a date field in Elasticsearch? Update the index template used so that the field in question is recognized as a date type with the format you've picked.

But why does this even matter?

---

<div class="post-metadata">

### Author: ![yash\_mangla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yash_mangla/32/25889_2.png) [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)
#### Post date: [June 2, 2017, 7:45am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/5 "2017-06-02T07:45:36Z")

</div>

yeah @magnusbaeck

> But why does this even matter?

i want fields as a date type.

It would be great help if you can show an example snippet of code by using above scenario for converting fields date type.  
and yeah,  
These event matter for me because when i do same thing with both kind of logs then it have standard timestamp format at one place for both timestamp in consolidated form so i can visualize these thing easily in kibana dashboard and it would be helpful to understand end user.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 8:19am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/6 "2017-06-02T08:19:25Z")

</div>

> i want fields as a date type.

I meant why it matters what exact format you store the timestamps as.

> These event matter for me because when i do same thing with both kind of logs then it have standard timestamp format at one place for both timestamp in consolidated form so i can visualize these thing easily in kibana dashboard and it would be helpful to understand end user.

The whole point of the date filter is that it converts different kinds of timestamps into a single format that Elasticsearch happens to understand and treat correctly out of the box. It's not clear to me why you were overcomplicating things by insisting on a different date format that ES doesn't understand without extra work.

---

<div class="post-metadata">

### Author: ![yash\_mangla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yash_mangla/32/25889_2.png) [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)
#### Post date: [June 2, 2017, 11:01am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/7 "2017-06-02T11:01:30Z")

</div>

Hi @magnusbaeck  
exact format is not matter but it should be any standard format which is same for both different timestamp which would be store in elasticsearch.

I just want how can i acheive this ?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 11:22am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/8 "2017-06-02T11:22:33Z")

</div>

As I said, use the date filter.

---

<div class="post-metadata">

### Author: ![yash\_mangla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yash_mangla/32/25889_2.png) [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)
#### Post date: [June 2, 2017, 11:27am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/9 "2017-06-02T11:27:49Z")

</div>

Thanks @magnusbaeck

i had another question i created another topic for this.  
can you revert on this topic and point out to me in right direction.

[what-would-be-the-filetype-for-our-custom-patterns-in-pattern-directory](https://discuss.elastic.co/t/what-would-be-the-filetype-for-our-custom-patterns-in-pattern-directory/88029)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2017, 11:28am UTC](https://discuss.elastic.co/t/how-to-convert-existing-timestamp-format-to-another-timestamp-format-using-logstash/87889/10 "2017-06-30T11:28:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.