# How to Create Graph using log file timestamp instead of kibana time

**URL:** <https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322>\
**Category:** Kibana\
**Tags:** elastic-stack-graph\
**Created:** [November 27, 2018, 10:49am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322 "2018-11-27T10:49:58Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [November 27, 2018, 10:49am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/1 "2018-11-27T10:49:58Z")

</div>

Hi Experts,

Please tell me how to Create a Graph using log file timestamp instead of Kibana time option.

Like in below screenshot named "image 1", I want to use logfile timestamp instead of "date histogram" in aggregation.

When I am choosing the Term keyword in aggregation that time I am not able to choose the "interval" option like daily basis/hourly basis etc, as showin in screenshot image 2.

Please advise what are the option to make the graph.

Image 1:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a0e61e9d436c5b5d1db8f774cff17eca5ea5c45.png)

Image 2:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c92598d701f0186536e780e4b4c38e5b102c606d.png)

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [November 28, 2018, 5:41am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/2 "2018-11-28T05:41:20Z")

</div>

msg\_timestamp is better to store in index as date datatype  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)

this approach as you printscreen would also work but it will show only 5 values (see SIZE 5) you could update it. But since it is not date type it will not be sorted as date but as string .

You shoult set correct datatype during insertion of the data into elasticsearch.  
I use templates it sets the correct mapping for matched imports  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html)  
within template just set  
"msg\_timestamp": {  
"type": "date"}

you can create 1 time mapping but templates is applied for every new index of the same type created

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [November 28, 2018, 12:32pm UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/3 "2018-11-28T12:32:56Z")

</div>

Hi Petr,

Thank you for the concern.

Can you please share your logstash filter conf file.

Regards  
Sanchit Gupta

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [December 6, 2018, 4:54am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/4 "2018-12-06T04:54:50Z")

</div>

Hi Petr,

I need your more help on this, i tried your advised way but not successing. can you please help me to setup/

Regards  
San

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [December 6, 2018, 5:38am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/5 "2018-12-06T05:38:17Z")

</div>

Have you created the mappings?

Please share your logstash conf in order to fix it.  
I suppose you should use date function some of this example  
in case target is not defined it automatically targets it @timestamp

> input {  
> file {  
> #some file or directory conf  
> }}
> 
> filter {
> 
> #Example1  
> date {  
> match =\> ["msg\_timestamp", "UNIX\_MS"]  
> timezone =\> "Etc/UTC"  
> target =\> "msgtime"  
> remove\_field =\> ["timestamp"]  
> }
> 
> #example2  
> date {  
> match =\> ["msg\_timestamp", "ISO8601"]  
> target =\> "msg\_timestamp"  
> }
> 
> #example 3  
> date {  
> match =\> ["msg\_timestamp", "YYYYMMddHHmmss"]  
> timezone =\> "Etc/UTC"  
> }
> 
> #example 4  
> date {  
> match =\> ["msg\_timestamp", "YYYYMMddHHmmss"]  
> timezone =\> "Etc/UTC"  
> target =\> "msg\_timestamp"  
> }  
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["127.0.0.1:9200"]  
> index =\> "test"  
> }
> 
> ```
> stdout { }
> 
> ```
> 
> }

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [December 6, 2018, 5:39am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/6 "2018-12-06T05:39:57Z")

</div>

check the reference guide  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html)

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [December 6, 2018, 5:41am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/7 "2018-12-06T05:41:51Z")

</div>

Hi Petr,

as per your advise i tried to create one template as shown below:  
after creation of this template i am facing the errors and even after deletion of this template still not working for me and even while creating the "Create index pattern" in kibana dashboard still i am getting the field named "created\_at"   
Create Template command:

```
PUT _template/template_1
{
  "index_patterns": ["admin*", "filebeat*"],
  "settings": {
	"number_of_shards": 1
  },
  "msg_timestamp": {
   "type": "date"},
  "mappings": {
	"_doc": {
	  "_source": {
		"enabled": false
	  },
	  
	  "properties": {
		"host_name": {
		  "type": "keyword"
		},
		"created_at": {
		  "type": "date",
		  "format": "EEE MMM dd HH:mm:ss Z YYYY"
		}
	  }
	}
  }
}

```

Delete Templete Command:  
DELETE /\_template/template\_1

Errors:

[2018-12-06T11:07:36,948][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"admin\_filebeat-2018.12.06", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x59ec1d48], :response=\>{"index"=\>{"\_index"=\>"admin\_filebeat-2018.12.06", "\_type"=\>"doc", "\_id"=\>"exIFgmcBLCbRLhh1fER2", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Rejecting mapping update to [admin\_filebeat-2018.12.06] as the final mapping would have more than 1 type: [\_doc, doc]"}}}}

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [December 6, 2018, 5:47am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/8 "2018-12-06T05:47:05Z")

</div>

Hi ,

I am just using the below code in logstash:

input:

```
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

```

filter:

```
filter {
if [message] !~ /^####/ {
	drop {
	}
}

grok {
	match => { "message" => "\#\#\#\#\<%{DATA:msg_timestamp}\> \<%{DATA:msg_severity}\> \<%{DATA:msg_subsystem}\>%{GREEDYDATA:msg_details}" }
}
}

```

output:

```
output {
elasticsearch {
hosts => ["xx.xxx.xx.xxx:9200"]
sniffing => true
manage_template => false
index => "admin_%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [December 6, 2018, 6:23am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/9 "2018-12-06T06:23:33Z")

</div>

Hi petr,

i tried to add filter but still not working for me.

```
  filter {
  if [message] !~ /^####/ {
	drop {
	}
  }

  grok {
	match => { "message" => "\#\#\#\#\<%{DATA:msg_timestamp}\> \<%{DATA:msg_severity}\> \<%{DATA:msg_subsystem}\>%{GREEDYDATA:msg_details}" }
  }
}
filter {
date {
match => ["msg_timestamp", "YYYYMMddHHmmss"]
timezone => "Etc/UTC"
target => "msg_timestamp"
}
}

```

output of msg\_timestamp: Nov 12, 2018 1:42:03 PM CST

Please check and advise .

---

<div class="post-metadata">

**Author:** ![bryan\_stuhlsatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_stuhlsatz/32/49123_2.png) [@bryan\_stuhlsatz](https://discuss.elastic.co/u/bryan_stuhlsatz)\
**Post date:** [December 6, 2018, 7:35am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/10 "2018-12-06T07:35:05Z")

</div>

I doubt that your match is working as that pattern is unusual with everything strung together. "YYYYMMddHHmmss"

Regardless, an issue you have is multiple mapping types. You have to fix that first. Just delete your index and you should be fine, and re-index your data. logstash uses "doc" and your mapping uses "\_doc."

---

<div class="post-metadata">

**Author:** ![Sanchit\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanchit_gupta/32/38108_2.png) [@Sanchit\_Gupta](https://discuss.elastic.co/u/Sanchit_Gupta)\
**Post date:** [December 6, 2018, 7:53am UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/11 "2018-12-06T07:53:50Z")

</div>

Hi Brynan,

i have deleted my index.

I am new to elk so understanding, where u are talking about "\_doc. "

---

<div class="post-metadata">

**Author:** ![bryan\_stuhlsatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_stuhlsatz/32/49123_2.png) [@bryan\_stuhlsatz](https://discuss.elastic.co/u/bryan_stuhlsatz)\
**Post date:** [December 6, 2018, 8:09pm UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/12 "2018-12-06T20:09:42Z")

</div>

Hello Sanchit. "\_doc" is what you posted earlier in your reply. Your conflicting doc types was the first error you were receiving. It is in the error you posted. Now that you have deleted your index, the next time you ingest data, it should bring in the data, and not have errors. The next step is to then start working on your date ingestion issue. Petr gave some excellent examples of how dates can be processed to match a pattern, and then ingest that field as a data type of date. Once you achieve that, then you should be able to do the queries that you want to.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 3, 2019, 8:09pm UTC](https://discuss.elastic.co/t/how-to-create-graph-using-log-file-timestamp-instead-of-kibana-time/158322/13 "2019-01-03T20:09:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.