# How to override @timestamp field by elasticsearch pipeline

**URL:** <https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262>\
**Category:** Elasticsearch\
**Created:** [December 12, 2017, 7:53am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262 "2017-12-12T07:53:54Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Augustus1995](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Augustus1995](https://discuss.elastic.co/u/Augustus1995)\
**Post date:** [December 12, 2017, 7:53am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/1 "2017-12-12T07:53:54Z")

</div>

Hi guys

I use filebeat to collect logs and send to elasticesearch directly. And the log contain the time such as

```
2017-12-12 15:41:30.000 [pool-8-thread-18] DEBUG c.h.i.c.m.servicemanager.task.PassiveNodeHbTask - PassiveNodeHbTask started

```

So i want to use the time in the log to override **@timestamp** and i don't want to use logstash

I have difined a pipeline:

```
   PUT _ingest/pipeline/my-pipeline-05
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:timestamp}"]
      }
    }
  ]
}

```

and i use this pipeline, but it didn't override **@timestamp** it create another field name **timestamp**

here is json

> ```
> {
> "_index": "tocc-2017.12.12",
> "_type": "doc",
> "_id": "rLe0SWAB2D7R7OG8M7Fc",
> "_version": 1,
> "_score": null,
> "_source": {
> "@timestamp": "2017-12-12T07:50:33.475Z",
> "log": {
> "level": "temp",
> "source": "temp",
> "type": "JAVA LOG"
> },
> "prospector": {
> "type": "log"
> },
> "source": "D:\\Program Files (x86)\\iVMS8600-TOCC\\log\\cms\\debug.log",
> "message": "2017-12-12 15:50:31.045 [pool-8-thread-16] DEBUG c.h.cms.cache.core.memory.SimpleMemoryCache - get key [#0#service_info_cache_version],value is:d1f9cef1-c406-4c27-9d8e-efadb92d5a1a",
> "timestamp": "2017-12-12 15:50:31.045"
> },
> "fields": {
> "@timestamp": [
> "2017-12-12T07:50:33.475Z"
> ]
> },
> "sort": [
> 1513065033475
> ]
> }
> 
> ```

So what can i do to override the **@timestamp** field

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 12, 2017, 8:06am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/2 "2017-12-12T08:06:10Z")

</div>

Use the `date` processor.

---

<div class="post-metadata">

**Author:** ![Augustus1995](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Augustus1995](https://discuss.elastic.co/u/Augustus1995)\
**Post date:** [December 12, 2017, 9:02am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/3 "2017-12-12T09:02:43Z")

</div>

thanks i have already try ,and the elasticsearch and filebeat didn't show any errors

But the i could get any message if i use the pipeline below

```
PUT _ingest/pipeline/my-pipeline-05
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:logatime}"]
      },
      "date" : {
        "field" : "logatime",
        "target_field" : "@timestamp",
        "formats" : ["YYYY-MM-DD HH:mm:ss.SSS"]
      }
    }
  ]
}

```

And if i didn't use **date** processor

I could get the message.

So am i mistake something

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 12, 2017, 12:13pm UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/4 "2017-12-12T12:13:20Z")

</div>

You configuration parameters for the `date` filter does not seem to match what is described in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-options), so I would recommend fixing that.

---

<div class="post-metadata">

**Author:** ![Augustus1995](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Augustus1995](https://discuss.elastic.co/u/Augustus1995)\
**Post date:** [December 12, 2017, 1:08pm UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/5 "2017-12-12T13:08:35Z")

</div>

I didn't use logstash , in fact i use elasticsearch's pipelines and i follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html).

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 12, 2017, 1:28pm UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/6 "2017-12-12T13:28:36Z")

</div>

Indeed. Not sure how I misread that. I would recommend that you compare the pattern against the docs and make sure all parts are specified using the correct case.

---

<div class="post-metadata">

**Author:** ![Augustus1995](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Augustus1995](https://discuss.elastic.co/u/Augustus1995)\
**Post date:** [December 12, 2017, 1:38pm UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/7 "2017-12-12T13:38:47Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> case

OK i will try ,and rewrite the date filter. I mean i didn't use logstash, and i misunderstand because i used believe maybe there have some different between the logstash's pipeline and elasticsearch's pipeline

---

<div class="post-metadata">

**Author:** ![Augustus1995](https://avatars.discourse-cdn.com/v4/letter/a/7cd45c/32.png) [@Augustus1995](https://discuss.elastic.co/u/Augustus1995)\
**Post date:** [December 13, 2017, 3:36am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/8 "2017-12-13T03:36:14Z")

</div>

Ok i find where is problem

Firstly, i need to use correct case

Second if you still could not find message , it could be you didn't set **timezone** , you need to set **timezone** and the problem would be solved

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2018, 3:36am UTC](https://discuss.elastic.co/t/how-to-override-timestamp-field-by-elasticsearch-pipeline/111262/9 "2018-01-10T03:36:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.