# How can i use the timestamp in the source file rather than the timestamp generate automatically

**URL:** <https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374>\
**Category:** Beats\
**Created:** [January 6, 2018, 1:48pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374 "2018-01-06T13:48:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![yigene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yigene/32/65553_2.png) [@yigene](https://discuss.elastic.co/u/yigene)\
**Post date:** [January 6, 2018, 1:48pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/1 "2018-01-06T13:48:36Z")

</div>

i am trying to use FileBeat directly parse the log file and send to elastic search. here is the example of the data  
2018-01-06 20:13:22 sending the number 7496 to outchannel  
2018-01-06 20:13:22 sending the number 7496 to inchannel  
2018-01-06 20:13:23 sending the number 7497 to outchannel  
2018-01-06 20:13:23 sending the number 7497 to inchannel  
2018-01-06 20:13:24 sending the number 7498 to outchannel  
2018-01-06 20:13:24 sending the number 7498 to inchannel  
2018-01-06 20:13:25 sending the number 7499 to outchannel  
2018-01-06 20:13:25 sending the number 7499 to inchannel  
2018-01-06 20:13:26 sending the number 7500 to outchannel  
2018-01-06 20:13:26 sending the number 7500 to inchannel

so i use the grok pipeline and the data can load to elasticsearch and below is the output  
{  
"took": 3,  
"timed\_out": false,  
"\_shards": {  
"total": 3,  
"successful": 3,  
"failed": 0  
},  
"hits": {  
"total": 2,  
"max\_score": 7.06019,  
"hits": [  
{  
"\_index": "mlsbeat-6.1.0-2018.01.06",  
"\_type": "doc",  
"\_id": "AWDLmoWt8ZoYLzM4Ts54",  
"\_score": 7.06019,  
"\_source": {  
"@timestamp": "2018-01-06T13:13:08.176Z",  
"offset": 235527,  
"beat": {  
"hostname": "gene-VirtualBox",  
"name": "gene-VirtualBox",  
"version": "6.1.0"  
},  
"prospector": {  
"type": "log"  
},  
"source": "/home/devops/mls-log.log",  
"message": "2018-01-06 20:05:01 sending the number 7000 to outchannel ",  
"rttype": "outchannel ",  
"timestamp": "2018-01-06 20:05:01",  
"trackingid": "7000"  
}  
},  
{  
"\_index": "mlsbeat-6.1.0-2018.01.06",  
"\_type": "doc",  
"\_id": "AWDLmoWt8ZoYLzM4Ts55",  
"\_score": 7.00971,  
"\_source": {  
"@timestamp": "2018-01-06T13:13:08.176Z",  
"offset": 235585,  
"beat": {  
"hostname": "gene-VirtualBox",  
"name": "gene-VirtualBox",  
"version": "6.1.0"  
},  
"prospector": {  
"type": "log"  
},  
"source": "/home/devops/mls-log.log",  
"message": "2018-01-06 20:05:01 sending the number 7000 to inchannel ",  
"rttype": "inchannel ",  
"timestamp": "2018-01-06 20:05:01",  
"trackingid": "7000"  
}  
}  
]  
}  
}

the @timestamp by default is the system time, but acutally i want to use the timestamp in the timestamp, in Kibana can not choose timestamp when create the index pattern hence not able to show in the dashboard. i understand the reason is that timestamp is string type not a date . is there anyway can easily convert it to date type without using logstash?

---

<div class="post-metadata">

**Author:** ![yigene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yigene/32/65553_2.png) [@yigene](https://discuss.elastic.co/u/yigene)\
**Post date:** [January 6, 2018, 1:50pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/2 "2018-01-06T13:50:20Z")

</div>

add the pipeline code

PUT \_ingest/pipeline/inspectionmls  
{  
"description" : "Convert mls rt log to indexed data",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["(?%{TIMESTAMP\_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]  
}  
}  
],  
"on\_failure" : [  
{  
"set" : {  
"field" : "error",  
"value" : " - Error processing message - "  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 6, 2018, 2:05pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/3 "2018-01-06T14:05:07Z")

</div>

Extract the event timestamp into a separate field in the pipeline and the use a [date processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html) to parse it.

---

<div class="post-metadata">

**Author:** ![yigene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yigene/32/65553_2.png) [@yigene](https://discuss.elastic.co/u/yigene)\
**Post date:** [January 6, 2018, 2:22pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/4 "2018-01-06T14:22:50Z")

</div>

it can not parse to Date format...

PUT \_ingest/pipeline/inspectionmls  
{  
"description" : "Convert mls rt log to indexed data",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["(?%{TIMESTAMP\_ISO8601}) sending the number %{DATA:trackingid} to %{DATA:rttype}$"]  
},  
"date": {  
"field": "timestamp",  
"formats": ["ISO8601"]  
}  
}  
]  
}

error:

{  
"docs": [  
{  
"error": {  
"root\_cause": [  
{  
"type": "exception",  
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"header": {  
"processor\_type": "date"  
}  
}  
],  
"type": "exception",  
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""  
}  
}  
},  
"header": {  
"processor\_type": "date"  
}  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 6, 2018, 2:26pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/5 "2018-01-06T14:26:45Z")

</div>

You need to change your grok expression to store the timestamp in a variable, e.g. `timestamp`, the same way you capture `trackingid` and `rttype`. Then you can apply the date processor to that field.

---

<div class="post-metadata">

**Author:** ![yigene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yigene/32/65553_2.png) [@yigene](https://discuss.elastic.co/u/yigene)\
**Post date:** [January 6, 2018, 2:33pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/6 "2018-01-06T14:33:06Z")

</div>

thanks..

but actually that already stored in timestamp variable. i feel is the date processor can not convert the string to date format hence throw out the exception

just tried again, the same error.

PUT \_ingest/pipeline/inspectionmls  
{  
"description" : "Convert mls rt log to indexed data",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["%{TIMESTAMP\_ISO8601:processtime} sending the number %{DATA:trackingid} to %{DATA:rttype}$"]  
},  
"date": {  
"field": "processtime",  
"formats": ["ISO8601"]  
}  
}  
]  
}

{  
"docs": [  
{  
"error": {  
"root\_cause": [  
{  
"type": "exception",  
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"header": {  
"processor\_type": "date"  
}  
}  
],  
"type": "exception",  
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "java.lang.IllegalArgumentException: unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "unable to parse date [2018-01-06 15:51:23]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": """Invalid format: "2018-01-06 15:51:23" is malformed at " 15:51:23""""  
}  
}  
},  
"header": {  
"processor\_type": "date"  
}  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 6, 2018, 2:38pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/7 "2018-01-06T14:38:37Z")

</div>

What does an event you previously got inserted with the incorrect timestamp look like? Can you try specifying the full date format instead of using `ISO8601` to see if that helps?

---

<div class="post-metadata">

**Author:** ![yigene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yigene/32/65553_2.png) [@yigene](https://discuss.elastic.co/u/yigene)\
**Post date:** [January 6, 2018, 2:44pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/8 "2018-01-06T14:44:53Z")

</div>

Thanks very much , after put below format it works!

```
  "date": {
    "field": "processtime",
    "formats": ["yyyy-MM-dd HH:mm:ss"]

      "@timestamp": "2018-01-06T15:51:23.000Z",
      "beat": {
        "hostname": "systemx",
        "name": "RestReviews",
        "version": "5.1.1"
      },
      "processtime": "2018-01-06 15:51:23",
      "rttype": "inchannels",
      "trackingid": "999711"
    },
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 27, 2018, 1:48pm UTC](https://discuss.elastic.co/t/how-can-i-use-the-timestamp-in-the-source-file-rather-than-the-timestamp-generate-automatically/114374/9 "2018-01-27T13:48:40Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.