# Parse logfile date into Kibana's timestamp

**URL:** <https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163>\
**Category:** Logstash\
**Created:** [April 12, 2017, 12:40pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163 "2017-04-12T12:40:00Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 12:40pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/1 "2017-04-12T12:40:00Z")

</div>

Hello, I have a date in log file like this one `2017-01-01 07:57:22` , I want to extract only the month and the day `01-01`, using a logstash filter, then use it as Kibana's timestamp.

here's how my logstash filter looks like:

```
input {
	tcp {
		port => 5000
		codec => multiline {
            pattern => "^(\s|{')"
            what => "previous"
        }
	}
}

filter {
	grok{
			 match => ["message", "'item_scraped_count': %{NUMBER:scraped:int}"]
			 match => ["message", "%{TIMESTAMP_ISO8601:timestamp}"]
	}

	date{
		     match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
		     target => "timestamp"
	}
}

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
	}
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 1:14pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/2 "2017-04-12T13:14:51Z")

</div>

You haven't really described what the problem is so it's hard to help. Have you configured the index pattern in Kibana to use `timestamp` as the timestamp field?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 1:18pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/3 "2017-04-12T13:18:23Z")

</div>

I edited the post, added the logstash config file.

> Have you configured the index pattern in Kibana to use timestamp as the timestamp field?

How can I achieve this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 1:25pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/4 "2017-04-12T13:25:38Z")

</div>

> I edited the post, added the logstash config file.

Yes, but that's not what I asked for. I understand what you want, but you're not telling us what you currently get. Is the `timestamp` field not populated with the parsed timestamp? Does it have the wrong timezone? Is it correctly populated but Kibana ignores it?

> How can I achieve this?

In the Kibana settings there's a dropdown for choosing the timestamp field for a particular index pattern. If you don't want to use the default `@timestamp` field (which Logstash is going to send anyway) you have to tell Kibana which field to use.

Why not stick with `@timestamp` until you're more comfortable with the stack?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 1:33pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/5 "2017-04-12T13:33:31Z")

</div>

> Is the timestamp field not populated with the parsed timestamp? Does it have the wrong timezone? Is it correctly populated but Kibana ignores it?

Yes, precisely.  
My extracted field exists with the correct values, but Kibana doesn't read it into its @timestamp field.

> In the Kibana settings there's a dropdown for choosing the timestamp field for a particular index pattern. If you don't want to use the default @timestamp field (which Logstash is going to send anyway) you have to tell Kibana which field to use.

Ah, I got it, but it only shows the default @timestamp field, maybe because my extracted field is parsed as a string not a date?

> Why not stick with @timestamp until you're more comfortable with the stack?

Problem is my extracted timestamp is different from the default one, the default one is the time when I indexed the logfile, I don't want this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 1:35pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/6 "2017-04-12T13:35:37Z")

</div>

> My extracted field exists with the correct values, but Kibana doesn't read it into its @timestamp field.

No, because you're storing the parsed timestamp in the `timestamp` field instead of in `@timestamp`. Remove the `target` option for your date filter so that you store it in `@timestamp`.

> Ah, I got it, but it only shows the default @timestamp field, maybe because my extracted field is parsed as a string not a date?

Yes, that's probably it.

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 1:44pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/7 "2017-04-12T13:44:16Z")

</div>

> [@magnusbaeck](#):
>
> No, because you're storing the parsed timestamp in the timestamp field instead of in @timestamp. Remove the target option for your date filter so that you store it in @timestamp.

I removed the target option line, but Kibana still parses them as seperate fields.

How my extracted field is parsed as a string? the date filter should return a date field, shouldn't it?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 1:46pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/8 "2017-04-12T13:46:43Z")

</div>

> I removed the target option line, but Kibana still parses them as seperate fields.

Please show your configuration and an example event from Kibana (produced by that configuration, of course).

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 1:54pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/9 "2017-04-12T13:54:29Z")

</div>

My logstash configuration:

```
filter {
	grok{
			 match => ["message", "'item_scraped_count': %{NUMBER:scraped:int}"]
			 match => ["message", "%{TIMESTAMP_ISO8601:timestamp}"]
	}

	date{
		     match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
	}
}

```

Kibana fields:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2adc064baed890bceec7ab26522b276c3dff9a6.png)

An example event:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/d/e/ded07494998324e2e0b9ff77f67291f2bdac6602.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 2:23pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/10 "2017-04-12T14:23:22Z")

</div>

The `_dateparsefailure` tag indicates that the date filter failed. Your Logstash logs will tell you why, but without looking I see that your date pattern isn't matching the input. You don't have any milliseconds in your `timestamp` field so you need to delete ",SSS" from your date pattern.

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 2:26pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/11 "2017-04-12T14:26:23Z")

</div>

Nice catch, how can I tweak my filters to extract only the month and day, like `2017-01-01 07:57:22` \>\> `01-01` ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 2:30pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/12 "2017-04-12T14:30:01Z")

</div>

I don't think timestamps without a year are supported. What would it even mean? How do you want to use such values?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 2:31pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/13 "2017-04-12T14:31:30Z")

</div>

Maybe if I am visualizing data within the same year.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 2:35pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/14 "2017-04-12T14:35:08Z")

</div>

That doesn't explain why you explicitly want to _remove_ the year from the timestamp.

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 2:37pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/15 "2017-04-12T14:37:20Z")

</div>

If all my events are within 2017 for example, I think it would be redundant and somehow boring to keep mentioning the year in the visualizations, don't you agree?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 2:45pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/16 "2017-04-12T14:45:44Z")

</div>

That's a visualization problem. You should still store the full date.

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 12, 2017, 2:57pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/17 "2017-04-12T14:57:27Z")

</div>

> so you need to delete ",SSS" from your date pattern.

still the same problem 😃

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 3:05pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/18 "2017-04-12T15:05:44Z")

</div>

Still getting `_dateparsefailure`? If yes, look in the Logstash log like I said. If no, what _do_ you get?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2017, 3:20pm UTC](https://discuss.elastic.co/t/parse-logfile-date-into-kibanas-timestamp/82163/19 "2017-05-10T15:20:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.