# @timestamp picked as string instead of date

**URL:** <https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876>\
**Category:** Elasticsearch\
**Created:** [November 23, 2017, 11:49am UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876 "2017-11-23T11:49:05Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 23, 2017, 11:49am UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/1 "2017-11-23T11:49:06Z")

</div>

Hi all,

I'm using Elasticsearch 6.0.0 and Filebeat 6.0.0 to forward and analyze some JMeter CSV test files.

I configured Filebeat to send some CSV data to an Elasticsearch instance through a pipeline, defined this way

```
PUT _ingest/pipeline/parse_test_csv
{
	"processors": [{
			"grok": {
				"field": "message",
				"patterns": ["%{INT:time},%{INT:elapsed},%{GREEDYDATA:label},%{INT:responseCode},%{DATA:responseMessage},%{GREEDYDATA:threadName},%{DATA:dataType},%{DATA:success},%{INT:bytes},%{INT:grpThreads},%{INT:allThreads},%{INT:latency}"]
			}
		}, {
			"date": {
				"field": "time",
				"formats": ["UNIX_MS"]
			}
		}, {
			"remove": {
				"field": ["threadName", "dataType", "bytes", "grpThreads", "allThreads"]
			}
		}, {
			"convert": {
				"field": "time",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "elapsed",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "label",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "responseCode",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "responseMessage",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "success",
				"type": "auto"
			}
		}, {
			"convert": {
				"field": "latency",
				"type": "auto"
			}
		}
	],
	"on_failure": [{
			"set": {
				"field": "error",
				"value": " - Error processing message - "
			}
		}
	]
}

```

As you can see the pipeline works with data formatted this way

`1511260490262,7241,A title,1000,Test successful,Tests 1-1,text,true,0,1,1,0`

The filebeat configuration regarding the elasticsearch output is simply this

```
output.elasticsearch:
  hosts: ["<host>:<port>"]
  pipeline: "parse_test_csv"

```

Now I would like to use `@timestamp` (which has the date time converted value of the `time` field as per the pipeline defined above) for the index pattern in Kibana. So I create a index pattern for `filebeat*`, but I am not able to select the `@timestamp` field as the time field for the pattern. It is recognized as string

 ![timestampfilebeat](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eef7aa2ce47a6bc5ccbe3801e0a7e38632cca5d6.png)

Can you help me finding what is wrong with my configuration? Where/how should I define the `@timestamp` field as a date in order to let me select it as the time field of the index pattern?

Thank you  
Giulio

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 23, 2017, 12:32pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/2 "2017-11-23T12:32:25Z")

</div>

I add here the topic's settings, with the dynamic mapping generated when the index was automatically created by ES

```
{
	"filebeat-6.0.0-2017.11.23": {
		"aliases": {},
		"mappings": {
			"doc": {
				"_meta": {
					"version": "6.0.0"
				},
				"dynamic_templates": [{
						"strings_as_keyword": {
							"match_mapping_type": "string",
							"mapping": {
								"ignore_above": 1024,
								"type": "keyword"
							}
						}
					}
				],
				"date_detection": false,
				"properties": {
					"@timestamp": {
						"type": "keyword",
						"ignore_above": 1024
					},
					"beat": {
						"properties": {
							"hostname": {
								"type": "keyword",
								"ignore_above": 1024
							},
							"name": {
								"type": "keyword",
								"ignore_above": 1024
							},
							"version": {
								"type": "keyword",
								"ignore_above": 1024
							}
						}
					},
					"elapsed": {
						"type": "long"
					},
					"fields": {
						"properties": {
							"app_name": {
								"type": "keyword",
								"ignore_above": 1024
							}
						}
					},
					"label": {
						"type": "keyword",
						"ignore_above": 1024
					},
					"latency": {
						"type": "long"
					},
					"message": {
						"type": "keyword",
						"ignore_above": 1024
					},
					"offset": {
						"type": "long"
					},
					"prospector": {
						"properties": {
							"type": {
								"type": "keyword",
								"ignore_above": 1024
							}
						}
					},
					"responseCode": {
						"type": "long"
					},
					"responseMessage": {
						"type": "keyword",
						"ignore_above": 1024
					},
					"source": {
						"type": "keyword",
						"ignore_above": 1024
					},
					"success": {
						"type": "boolean"
					},
					"time": {
						"type": "float"
					}
				}
			}
		},
		"settings": {
			"index": {
				"mapping": {
					"total_fields": {
						"limit": "10000"
					}
				},
				"refresh_interval": "5s",
				"number_of_shards": "1",
				"provided_name": "filebeat-6.0.0-2017.11.23",
				"creation_date": "1511439697431",
				"number_of_replicas": "1",
				"uuid": "JLvQ6CuoQyivdiuq53XcUA",
				"version": {
					"created": "6000099"
				}
			}
		}
	}
}

```

As you see the `@timestamp` field has the `keyword` data type assigned by ES, not the `date` type. So something went wrong in ES when it created the index dynamic type (or not?)

```
"@timestamp": {
	"type": "keyword",
	"ignore_above": 1024
}
```

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 23, 2017, 2:59pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/3 "2017-11-23T14:59:58Z")

</div>

Please, I'm stuck with this problem. Does anybody know what the problem could be?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 23, 2017, 3:53pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/4 "2017-11-23T15:53:38Z")

</div>

Please read

> [@About the Elasticsearch category](https://discuss.elastic.co/t/about-the-elasticsearch-category/21):
>
> The heart of the free and open Elastic Stack Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease. warning PLEASE READ THIS SECTION IF IT'S YOUR FIRST POST Some useful links: [elasticsearch reference guide](http://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)[elasticsearch user guide](http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html)[elasticsearch plugins](https://www.elastic.co/guide/en/elasticsearch/plugins/current/index.html)[elasticsearch cl…](https://www.elastic.co/guide/en/elasticsearch/client/index.html)

Specifically the "be patient" part.

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 23, 2017, 4:33pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/5 "2017-11-23T16:33:19Z")

</div>

Sorry, is not my intention to hurry you.

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 24, 2017, 3:58pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/6 "2017-11-24T15:58:06Z")

</div>

An update.

I tried disabling the pipeline at all, so the CSV document is sent to ES directly.

Again, the index gets created with the `@timestamp` field as `keyword`, so it seems that Filebeat creates the index in ES with the `@timestamp` field with type `keyword`, but having a regulard date time value in it like `2017-11-24T15:51:19.913Z`.

It's so strange because the internal `@timestamp` field is created by Filebeat/ES and it should be managed automatically with the `date` type (as stated [here](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-beat.html#_timestamp))

What do you think?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 24, 2017, 4:23pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/7 "2017-11-24T16:23:46Z")

</div>

When filebeat starts, I believe that it creates an index template that you might need to adapt.

What is the current templates you have? `GET _template`

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [November 24, 2017, 5:00pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/8 "2017-11-24T17:00:30Z")

</div>

I found the issue and fixed it.

"Simply" I disabled the default templating of Filebeat (the ones written in `fields.xml`) so Filebeat was creating the fields without a type mapping.  
In fact, in my trials I set the `setup.template.fields` property to a empty file. Now I set it to a custom file having the `beat` mapping section only and everything is working as expected.

Sorry for having created a thread for an issue which usually should not exist, but I hope this will be useful in the future for others.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 22, 2017, 5:00pm UTC](https://discuss.elastic.co/t/timestamp-picked-as-string-instead-of-date/108876/9 "2017-12-22T17:00:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.