# Parsing the date and sending to ElasticSearch **URL:** \ **Category:** Logstash\ **Created:** [April 27, 2017, 2:48pm UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883 "2017-04-27T14:48:23Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 27, 2017, 2:48pm UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/1 "2017-04-27T14:48:23Z")
Hello everyone, Great to be here 🙂 For hours am trying to parse a file from filebeat and send it to ES. It works, but the date field that is created in ES is set to string, instead of date. I am using (Mac): ES: 5.3.0 Filebeat: 5.3.0 Logstash: 5.3.0 **This is the line that is send from Filebeat:** > [2017-04-26 09:40:32] request.INFO: Matched route "home\_logged\_in". {"route\_parameters":{"\_controller":"AppBundle\Controller\HomeLoggedInController::showAction","\_locale":"de","\_route":"home\_logged\_in"},"request\_uri":"[https://something.com](https://something.com)"} This is the Logstash portion that parses this line: > if [@metadata][type] == "prod" or [@metadata][type] == "qaprod"{ > grok { > match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logdate}" } > } > date { > match =\> ["logdate", "ISO8601"] > . target =\> "logdate" > } > } Now, when I look in the Kibana for the logdate, I see it exactly like this 2017-04-26 09:40:32. Problem is it is set as string. I need it as date. Can you please help me? ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 27, 2017, 3:47pm UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/2 "2017-04-27T15:47:37Z")
I am running out of solutions. I have checked all similar problems in the forum to no avail. I have tried many different things, and it always ends up in ES as string. This is the last thing I tried: > ``` > if [@metadata][type] == "prod" or [@metadata][type] == "qaprod"{ > grok { > match => { "message" => "%{TIMESTAMP_ISO8601:logdate}" } > } > mutate { > convert => ["logdate", "string"] > add_field => { "pleasework" => "%{logdate}" } > } > date { > timezone => "Europe/Berlin" > match => ["pleasework", "ISO8601", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] > #target => "pleasework" > } > } > > ``` And of course, please work is still in string format. For all I care, it can be set to timestamp as well. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 5:37am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/3 "2017-04-28T05:37:35Z")
A string that has been successfully processed by the date filter should be recognized as a date by ES. However, this recognition only happens the first time the field is seen. What do the index's mappings look like? Please post the output of a get mapping API call. Please also post an example document (preferably via copy/paste from the JSON tab in Kibana). ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 28, 2017, 7:01am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/4 "2017-04-28T07:01:13Z")
Thank you very much for your time!. I really appreciate it. I always delete the index before sending new data to it. So it is always starting from scratch. Here is JSON copy from the current Kibana. Please note there are some other errors, as I need to apply more GROK filters to it. I will do that, as soon as I manage to fix this date issue. I really do not need a special date field, I am happy if I can map logdate to @timestamp field. From Kibana: > "type": "fesotprod", > "tags": [ > "\_jsonparsefailure", > "beats\_input\_codec\_json\_applied", > "\_dateparsefailure" > ], > "@timestamp": "2017-04-27T15:57:43.297Z", > "logdate": "2017-04-26 09:40:33", > "@version": "1", > "beat": { > "hostname": "C700893", > "name": "C700893", > "version": "5.3.0" > }, > "host": "C700893", > "fingerprint": "844563e8094c0c1810c04b3347155ad4f0082dff" > }, > "fields": { > "@timestamp": [ > 1493308663297 > ] > } Two sample string from the original log (log starts with the date): > 2017-04-26 09:40:33] security.DEBUG: Stored the security token in the session. {"key":"\_security\_secured\_area"} > [2017-04-26 09:50:42] request.INFO: Matched route "home\_logged\_in". {"route\_parameters":{"\_controller":"AppBundle\Controller\HomeLoggedInController::showAction","\_locale":"de","\_route":"home\_logged\_in"},"request\_uri":"[https://qa.someserver.de/de/home](https://qa.someserver.de/de/home)"} Get mapping call on the index after was created (logdate is the problematic field): > { > "fesotprod": { > "mappings": { > "fesotprod": { > "properties": { > "@timestamp": { > "type": "date" > }, > "@uuid": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "@version": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "beat": { > "properties": { > "hostname": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "name": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "version": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > } > } > }, > "fingerprint": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "host": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "input\_type": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "logdate": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "message": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "offset": { > "type": "long" > }, > "source": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "tags": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > }, > "type": { > "type": "text", > "fields": { > "keyword": { > "type": "keyword", > "ignore\_above": 256 > } > } > } > } > } > } > } > } Thank you very much. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 7:26am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/5 "2017-04-28T07:26:25Z")
I think you have two options: - Fix your date filter so that it works and stores the result in `logdate` (if that really is where you want the timestamp stored). - Adjust the mappings of your index so that "2017-04-26 09:40:33" is recognized as a date. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 28, 2017, 8:01am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/6 "2017-04-28T08:01:56Z")
Hello Magnus, If I use this: > grok { > match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logdate}" } > } > date { > timezone =\> "Europe/Berlin" > match =\> ["logdate", "ISO8601", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] > } For these log lines: > 2017-04-26 09:40:33] security.DEBUG: Stored the security token in the session. {"key":"securitysecured\_area"} > [2017-04-26 09:50:42] request.INFO: Matched route "home\_logged\_in". {"route\_parameters":{"controller":"AppBundle\Controller\HomeLoggedInController::showAction","locale":"de","route":"homelogged\_in"},"request\_uri":"[https://qa.someserver.de/de/home](https://qa.someserver.de/de/home)"} The way I understand it, it should overwrite the @timestamp. But it is not happening. I am still getting it as a string in ES. You mentioned that I should fix my filter. I am really out of ideas now. Would you provide me some guidance? 🙂 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 8:11am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/7 "2017-04-28T08:11:23Z")
Are you still getting the `_dateparsefailure` tag? If yes, look in your log for clues about why the date filter fails. It could be that the ISO8601 pattern doesn't match "2017-04-26 09:40:33" (because you have no "T" between the date and the time). A more exact pattern (similar to the second pattern you've listed) will definitely work. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 28, 2017, 8:46am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/8 "2017-04-28T08:46:38Z")
I think I am on correct way. In my case, this is the pattern that should work: > %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})? Now how do I use it in my case? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 8:53am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/9 "2017-04-28T08:53:39Z")
That's a grok pattern and we're talking about the date filter. Your second date pattern is "yyyy-MM-dd'T'HH:mm:ss.SSSZ" which is very close to what your `logdate` field looks like. It just needs a small adjustment. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 28, 2017, 9:00am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/10 "2017-04-28T09:00:20Z")
I think I have constructed the pattern that works. Here it is: > (?\>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?) I am confused now, how do I use it to extract the date in my case. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 9:14am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/11 "2017-04-28T09:14:46Z")
For the last time, **stop modifying your grok filter**. It was working earlier. Focus on the date filter. It's nearly correct but it needs a small adjustment to match what's in `logdate` (e.g. "2017-04-26 09:40:33"). Over and out. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [April 28, 2017, 9:26am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/12 "2017-04-28T09:26:40Z")
For this string: [2017-04-26 15:23:52] When I use this: > if [@metadata][type] == "fesotprod" or [@metadata][type] == "qafesotprod"{ > grok { > match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logdate}" } > } > date { > #timezone =\> "Europe/Berlin" > match =\> ["logdate", "yyyy-MM-dd HH:mm:ss"] > } > } I am getting a field created in ES. The field name is logdate, and it has value of 2017-04-26 15:23:52. But it is string. Sorry if I piss you off somehow. Was not my intention. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 28, 2017, 11:06am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/13 "2017-04-28T11:06:22Z")
Unless told otherwise with the `target` option the date filter stores the parsed timestamp in the `@timestamp` field. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2017, 11:11am UTC](https://discuss.elastic.co/t/parsing-the-date-and-sending-to-elasticsearch/83883/14 "2017-05-26T11:11:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.