# Using index template to read a field as date

**URL:** <https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 21, 2018, 8:14am UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322 "2018-08-21T08:14:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonathan\_Tan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_tan/32/83546_2.png) [@Jonathan\_Tan](https://discuss.elastic.co/u/Jonathan_Tan)\
**Post date:** [August 21, 2018, 8:14am UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/1 "2018-08-21T08:14:14Z")

</div>

Hi,

I'm attempting to take a JSON-based log, and use Filebeat to stick it directly into Elasticsearch. I'm trying - at this point - not to include Logstash yet, AND also attempting to get a better understanding of what Filebeat is capable of.

Whilst I can currently get the log loaded into Elasticsearch, I can't seem to get my timestamp field to be recognised as a date. I'm aware that Filebeat will add a `@timestamp` property that reflects when it read the log item. I don't mind that, but I also want my own timestamp property to be usable for time series analysis.

Some sample log items. I'm looking to use the `eventTime` property as my timestamp field.

```auto
{"eventType":"search","searchTerm":"sauce","user":175362,"eventTime":"2018-08-21T15:42:29+1000"}
{"eventType":"search","searchTerm":"goats milk","user":138297,"eventTime":"2018-08-21T15:42:29+1000"}
{"eventType":"search","searchTerm":"potatoes","user":140003,"eventTime":"2018-08-21T15:42:29+1000"}

```

This is what I've done, and none of them have succeeded in having the `eventTime` field registered as a `date` type.

Partial of the `filebeat.yml` file

```auto
output.elasticsearch:
  hosts: ['elasticsearch:9200']
  index: "search-metrics-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.template.name: "search-metrics-filebeat"
setup.template.pattern: "search-metrics-filebeat-*"
setup.template.fields: "/usr/share/filebeat/events-fields.yml"
setup.template.overwrite: true

```

for the `events-field.yml` file, it's contents were extracted the mapping from my index, I then used a JSON -\> YAML converter (I've got NO idea if this is the right thing to do or not...), and then I made the following modifications:

- set `date_detection: true`
- set `dynamic_date_formats: ["YYYY-MM-DDTHH:mm:ssZZ"]`
- (and finally when I decided to try the brute force approach) I added an `eventTimeField` under `dynamic_templates` that matched on the property name of `eventTime`

```auto
  doc:
    _meta:
      version: 6.3.2
    date_detection: true
    dynamic_date_formats: ["YYYY-MM-DDTHH:mm:ssZZ"]
    dynamic_templates:
    - eventTimeField:
        match_mapping_type: string
        match: "eventTime"
        mapping:
          type: "date"
    - fields:
        mapping:
          type: keyword
        match_mapping_type: string
        path_match: fields.*

```

I also tried, in the `properties` section, at the same level as the `apache2` and `docker` entries, adding a `date` property like that:

```auto
      date:
        type: "date"
        format: "YYYY-MM-DDTHH:mm:ssZZ||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"

```

As you might guess, I'm basically taking bits and pieces from where-ever and trying to figure out if it works or not...

Any chance of some help please?

Thank you

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 21, 2018, 6:35pm UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/2 "2018-08-21T18:35:03Z")

</div>

@Jonathan_Tan as you have note Filebeat automatically add a new `@timestamp` on every line that it read. By default this the time when the line was read, which is usually not what you want.

Since your original data is already a structured JSON document I think you should use an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html) to configure some logic on the ES side to parse the date correctly. To do that you will need the [date processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html) and you will have to change your [elasticsearch output configuration](https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html)

The above is better because this will make the `@timestamp` field reflect the real time of the event and that field is the default field for Kibana.

If you want to keep the `time read` you can use the [rename processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/rename-processor.html) before the date processor.

---

<div class="post-metadata">

**Author:** ![Jonathan\_Tan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_tan/32/83546_2.png) [@Jonathan\_Tan](https://discuss.elastic.co/u/Jonathan_Tan)\
**Post date:** [August 21, 2018, 11:54pm UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/3 "2018-08-21T23:54:47Z")

</div>

Hi Pier,

Thank you for your suggestion. So just to confirm, you're essentially saying "let filebeat stream, and give the transformation to elastic search"?

I'll take a look at the `ingest` stuff in more detail. It does look straightforward too.

Thank you!

---

<div class="post-metadata">

**Author:** ![Jonathan\_Tan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_tan/32/83546_2.png) [@Jonathan\_Tan](https://discuss.elastic.co/u/Jonathan_Tan)\
**Post date:** [August 22, 2018, 4:38am UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/4 "2018-08-22T04:38:46Z")

</div>

Hi @pierhugues,

I took your advice, stripped out my existing filebeat config attempts, and used an Elasticsearch ingest pipeline instead. That _is_ easier and it did the trick.

That said, I've also come to realise that Elasticsearch isn't logging pipeline errors, and instead Filebeat is. And Filebeat isn't logging useful errors normally either, only in verbose mode. I see that there's already another discussion regarding Elasticsearch's error handling, so I might go make my two cents known there. 😉

For reference for anybody else, this is what I did using the Elasticsearch ingest to read an existing field, parse it as a timestamp, and then put it into the `@timestamp` field:

`PUT http://localhost:9200/_ingest/pipeline/search_event_pipeline`, with the following post body:

```auto
{
    "description": "Takes the eventTime field and turns it into a date field",
    "processors": [
        {
            "date": {
                "field": "eventTime",
                "target_field": "@timestamp",
                "formats": [
                    "YYYY-MM-DD'T'HH:mm:ssZZ",
                    "YYYY/MM/DD HH:mm:ssZZ"
                ]
            }
        }
    ],
    "on_failure": [
        {
            "set": {
                "field": "_index",
                "value": "failed-{{_index}}"
            }
        },
        {
            "set": {
                "field": "error",
                "value": "{{_ingest.on_failure_message}}"
            }
        }
    ]
}

```

The above `on_failure` property was to eventually handle the errors, and to - more importantly - pipe them into a separate index for me to assess.

In the future, I _might_ remove the original `eventTime` field as well, but I'd kept it there this time so I could make sure it was working... 😉

In the `filebeat.yml` file, my `elastic` output was turned into:

```auto
output.elasticsearch:
  hosts: ['elasticsearch:9200']
  index: "search-metrics-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
  pipeline: "search_event_pipeline"

```

Again, Pier, thanks for your help 😄

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 22, 2018, 12:30pm UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/5 "2018-08-22T12:30:59Z")

</div>

@Jonathan_Tan glad that it worked! Yeah we are trying to improve any errors coming from that 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2018, 12:31pm UTC](https://discuss.elastic.co/t/using-index-template-to-read-a-field-as-date/145322/6 "2018-09-19T12:31:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.