Mutate timestamp for nginx logs

Elastic Stack Logstash
1

Continuing the discussion from Keep Logstash from Crashing:

@dedemorton gave a workaround that prevents logstash from crashing, but i end up with a read-timestamp field with the literal string value "@timestamp" in the read_timestamp field.

So I am using this:

  mutate {
      add_field => { "stashed_time" => "%{[@timestamp]}" }
    }
    date {
      match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
    }

Was this a syntax change from 5.6 to 6.0.0 ? ( I am running 6.0.0-rc1 )

2

end up with a read-timestamp field with the literal string value "@timestamp" in the read_timestamp field.

Do you mean stashed_time field?

3

Hey @magnusbaeck, no my code with stashed_time is working fine.

I had the problem when I tried the suggestion that I referenced from the old thread:

mutate {
      add_field => { "read_timestamp" => "@timestamp" }
   }

I am continuing the thread now, to offer a solution that seems to work with 6.0 and to ask if there was a syntax change from 5.6.

Either way, this indicates that the docs need an update, as they are renaming the @timestamp field without adding a new field - which can lead to "no timestamp field errors"

I see that you also engaged in the previous thread, @magnusbaeck... In general, do you recommend to remove the date from the default index?

Thanks for the help, still finding my way into the world of ELK+beats

4 
  add_field => { "read_timestamp" => "@timestamp" }

Yeah, this won't work. You need %{@timestamp} or the equivalent %{[@timestamp]}.

Either way, this indicates that the docs need an update, as they are renaming the @timestamp field without adding a new field - which can lead to "no timestamp field errors"

Yes, that's a documentation bug.

In general, do you recommend to remove the date from the default index?

I recommend keeping the @timestamp field but making sure that it contains the timestamp when the event occurred.

5

Great! that was my instinct as well.

This code grok, mutate, date seems to accomplish that:

  if [type] == "nginx-access" {
    grok {
      match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{DATA:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
    }
    mutate {
      add_field => { "stashed_time" => "%{[@timestamp]}" }
    }
    date {
      match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
    }
  }

Any recommendations for improvement?

6

Yes, don't use the DATA pattern so much. Logstash ships with predefined patterns for HTTP logs and you can use them for inspiration.

7

Yes, sorry about the confusion. The example should have been add_field => { "read_timestamp" => "%{@timestamp}" }

I've already fixed the issue in 6.0 (update will be published soon) and will backport the change to 5.6.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.