# Problem with @timestamp time in indexes

**URL:** <https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 5, 2020, 11:47am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280 "2020-03-05T11:47:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 5, 2020, 11:47am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/1 "2020-03-05T11:47:42Z")

</div>

Hello, we have filebeat sending messages directly to elasticsearch index. And we found such problem, @timestamp in kibana view ( when we search) is different from timestamp we have in file on server( which use filebeat). Looks like elastic change timestamp - to time when document was indexed, not timestamp from original document( which could be found in file on server) How to fix that ? Because it's important for us to have correct messages order in time

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 5, 2020, 12:40pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/2 "2020-03-05T12:40:12Z")

</div>

Are you using a specific filebeat module?  
How your logs are parsed?  
Are you using an ingest pipeline?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 5, 2020, 9:06pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/3 "2020-03-05T21:06:12Z")

</div>

I use filebeat 7.4.2, logs are text files with json ojects on each row. I do not use igest pipline

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 6, 2020, 2:51am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/4 "2020-03-06T02:51:08Z")

</div>

So elasticsearch just index the json content as you are sending it.  
You need to define an ingest pipeline to rename the field which contains the event date to `@timestamp`.

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 6, 2020, 10:25am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/5 "2020-03-06T10:25:44Z")

</div>

So if we have such field named @timestamp in logs on server, will it be replaced? Beacause our event date fiels called @timestamp - default name ...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 6, 2020, 4:12pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/6 "2020-03-06T16:12:04Z")

</div>

Can you show a document?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 6, 2020, 8:21pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/7 "2020-03-06T20:21:41Z")

</div>

Here is docuent in index [https://take.ms/Ybmfy](https://take.ms/Ybmfy)

And here is document in log on server  
{"@timestamp":"2020-03-06T20:18:23.894670031Z","exportable":true,"level":"info","message":"xxx","report\_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","username":"\_eva....."}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 7, 2020, 11:40am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/8 "2020-03-07T11:40:22Z")

</div>

Could you run

```
GET INDEXNAME/_doc/ID

```

where

- `INDEXNAME` is the filebeat index name
- `ID` is the `_id` of the document

And share the output here.

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with `</>` icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 8, 2020, 10:44am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/9 "2020-03-08T10:44:39Z")

</div>

{"\_index":"xxx-2020.02.21-000001","\_type":"\_doc","\_id":"GG5\_sXABULG93gUD3XMn","\_version":1,"\_seq\_no":846086,"\_primary\_term":1,"found":true,"\_source":{"@timestamp":"2020-03-06T20:18:29.973Z","log":{"file":{"path":"xxx.log"},"offset":14310474576},"report\_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","input":{"type":"log"},"ecs":{"version":"1.1.0"},"exportable":true,"level":"info","message":"xxx","host":{"name":"xxx"},"agent":{"hostname":"xxx","id":"448f24c9-d971-4558-8651-b57f65cc4de2","version":"7.4.2","type":"filebeat","ephemeral\_id":"9bf7ff2d-3c39-4e04-8b8d-d2fe300d4f3e"},"username":"\_eva"}}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 8, 2020, 11:59am UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/10 "2020-03-08T11:59:37Z")

</div>

I can't see any other timestamp in this document. So I don't understand what the problem is. Unless you removed important things in the example you shared?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 8, 2020, 12:15pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/11 "2020-03-08T12:15:08Z")

</div>

Nothing removed. And problem is that time stamps in index and in file are different.  
20:18:29 in index and 20:18:23 in file.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 8, 2020, 4:45pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/12 "2020-03-08T16:45:42Z")

</div>

Can you share the full json document without removing anything?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 8, 2020, 4:57pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/13 "2020-03-08T16:57:19Z")

</div>

I did not remove any fields, i just replace some text to xxx, is that important? ( there only index name, log filename, message, hostname and thats all )

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 8, 2020, 6:03pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/14 "2020-03-08T18:03:54Z")

</div>

Yes it is

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 8, 2020, 7:23pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/15 "2020-03-08T19:23:34Z")

</div>

```
{"_index":"trendhero-2020.02.21-000001","_type":"_doc","_id":"GG5_sXABULG93gUD3XMn","_version":1,"_seq_no":846086,"_primary_term":1,"found":true,"_source":{"@timestamp":"2020-03-06T20:18:29.973Z","log":{"file":{"path":"/var/log/report-generator/report-generator.log"},"offset":14310474576},"report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","input":{"type":"log"},"ecs":{"version":"1.1.0"},"exportable":true,"level":"info","message":"Get full report request","host":{"name":"1ps-api"},"agent":{"hostname":"1ps-api","id":"448f24c9-d971-4558-8651-b57f65cc4de2","version":"7.4.2","type":"filebeat","ephemeral_id":"9bf7ff2d-3c39-4e04-8b8d-d2fe300d4f3e"},"username":"_eva.vls_"}}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 8, 2020, 8:58pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/16 "2020-03-08T20:58:35Z")

</div>

So which field elasticsearch is supposed to use as a timestamp of the event?

I don't see anything that has been collected by filebeat which looks like a timestamp.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 8, 2020, 9:00pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/17 "2020-03-08T21:00:15Z")

</div>

Could you share the log line that has been collected by filebeat which corresponds to this Event in elasticsearch?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 9, 2020, 4:52pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/18 "2020-03-09T16:52:08Z")

</div>

```
{"@timestamp":"2020-03-06T20:18:23.894670031Z","exportable":true,"level":"info","message":"Get full report request","report_uuid":"bf0f8a56-4a46-4cf1-8706-491b3607eaf3","username":"_eva.vls_"}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 9, 2020, 5:04pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/19 "2020-03-09T17:04:01Z")

</div>

What is the filebeat configuration?

---

<div class="post-metadata">

**Author:** ![Vinnyard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinnyard/32/63886_2.png) [@Vinnyard](https://discuss.elastic.co/u/Vinnyard)\
**Post date:** [March 9, 2020, 5:20pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280/20 "2020-03-09T17:20:20Z")

</div>

```
filebeat.inputs:

- type: log
  enabled: true
  json.keys_under_root: true
  paths:
    - /var/log/report-generator/report-generator.log

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elk.trendhero.io:9200"]
  index: "trendhero-%{+yyyy.MM.dd}"

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "REMOVED"

setup.template:
  name: 'trendhero'
  pattern: 'trendhero-*'
  enabled: false

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "trendhero"
setup.ilm.pattern: "{now/d}-000001"

logging.level: info

processors:
 - drop_event:
       when:
           not:
               equals:
                   exportable: true
```

[Next page](https://discuss.elastic.co/t/problem-with-timestamp-time-in-indexes/222280.md?page=2)