# Mutate timestamp for nginx logs

**URL:** <https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814>\
**Category:** Logstash\
**Created:** [October 22, 2017, 6:17pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814 "2017-10-22T18:17:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gotjoshua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gotjoshua/32/23102_2.png) [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\
**Post date:** [October 22, 2017, 6:17pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/1 "2017-10-22T18:17:35Z")

</div>

Continuing the discussion from [Keep Logstash from Crashing](https://discuss.elastic.co/t/keep-logstash-from-crashing/90392/6):

@dedemorton gave a workaround that prevents logstash from crashing, but i end up with a read-timestamp field with the literal string value "@timestamp" in the read\_timestamp field.

So I am using this:

```
  mutate {
      add_field => { "stashed_time" => "%{[@timestamp]}" }
    }
    date {
      match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
    }

```

Was this a syntax change from 5.6 to 6.0.0 ? ( I am running 6.0.0-rc1 )

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 23, 2017, 5:22am UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/2 "2017-10-23T05:22:50Z")

</div>

> end up with a read-timestamp field with the literal string value "@timestamp" in the read\_timestamp field.

Do you mean `stashed_time` field?

---

<div class="post-metadata">

**Author:** ![gotjoshua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gotjoshua/32/23102_2.png) [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\
**Post date:** [October 23, 2017, 8:14am UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/3 "2017-10-23T08:14:37Z")

</div>

Hey @magnusbaeck, no my code with stashed\_time is working fine.

I had the problem when I tried the suggestion that I referenced from the old thread:

```
mutate {
      add_field => { "read_timestamp" => "@timestamp" }
   }

```

I am continuing the thread now, to offer a solution that seems to work with 6.0 and to ask if there was a syntax change from 5.6.

Either way, this indicates that [the docs need an update](https://www.elastic.co/guide/en/logstash/6.0/logstash-config-for-filebeat-modules.html#parsing-nginx), as they are renaming the @timestamp field without adding a new field - which can lead to "no timestamp field errors"

I see that you [also engaged in the previous thread](https://discuss.elastic.co/t/keep-logstash-from-crashing/90392/4?u=gotjoshua), @magnusbaeck... In general, do you recommend to remove the date from the default index?

Thanks for the help, still finding my way into the world of ELK+beats

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 23, 2017, 8:41am UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/4 "2017-10-23T08:41:15Z")

</div>

> ```
> add_field => { "read_timestamp" => "@timestamp" }
> 
> ```

Yeah, this won't work. You need `%{@timestamp}` or the equivalent `%{[@timestamp]}`.

> Either way, this indicates that the docs need an update, as they are renaming the @timestamp field without adding a new field - which can lead to "no timestamp field errors"

Yes, that's a documentation bug.

> In general, do you recommend to remove the date from the default index?

I recommend keeping the `@timestamp` field but making sure that it contains the timestamp when the event occurred.

---

<div class="post-metadata">

**Author:** ![gotjoshua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gotjoshua/32/23102_2.png) [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\
**Post date:** [October 23, 2017, 9:06am UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/5 "2017-10-23T09:06:23Z")

</div>

> [@magnusbaeck](#):
>
> I recommend keeping the @timestamp field but making sure that it contains the timestamp when the event occurred.

Great! that was my instinct as well.

This code grok, mutate, date seems to accomplish that:

```
  if [type] == "nginx-access" {
    grok {
      match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{DATA:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
    }
    mutate {
      add_field => { "stashed_time" => "%{[@timestamp]}" }
    }
    date {
      match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
    }
  }

```

Any recommendations for improvement?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 23, 2017, 1:12pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/6 "2017-10-23T13:12:18Z")

</div>

Yes, don't use the DATA pattern so much. Logstash ships with predefined patterns for HTTP logs and you can use them for inspiration.

> <https://github.com/logstash-plugins/logstash-patterns-core/blob/v4.1.2/patterns/httpd>

---

<div class="post-metadata">

**Author:** ![dedemorton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dedemorton/32/84409_2.png) [@dedemorton](https://discuss.elastic.co/u/dedemorton)\
**Post date:** [October 23, 2017, 6:32pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/7 "2017-10-23T18:32:34Z")

</div>

Yes, sorry about the confusion. The example should have been `add_field => { "read_timestamp" => "%{@timestamp}" }`

I've already fixed the issue in 6.0 (update will be published soon) and will backport the change to 5.6.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2017, 6:32pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814/8 "2017-11-20T18:32:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.