How to convert Apache timestamp(HTTPDATE) to ISO8601 format log

Elastic Stack Logstash
1

Hi experts,

Currently I'm forwarding apache access log to output target with the configuration, and I'd like to convert current timestamp which is Apache timestamp format (HTTPDATE) to ISO8601 format and add it to output fields. Can you please guide me the way to achieve my goal?

input {
    file {
        path => "/var/log/apache2/access.log"
        start_position => "beginning"
    }
}

filter {
    mutate { replace => { "type" => "apache_access" } }
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

output {
    # for debug
    stdout { codec => rubydebug }
}
2

Your current configuration looks fine to me. What isn't working? Show us an example event.

3

@magnusbaeck
Thank you so much for your quick response! My explanation wasn't enough,
What I would like to achieve is to convert 'timestamp' field (format:"dd/MMM/yyyy:HH:mm:ss Z") to ISO8601 format ("YYYY-MM-DDThh:mm:ssZ"). Really appreciate If you could guide me how to archieve it.

4

Yes, I know what you want to achieve. What I don't know is what Logstash currently produces with your configuration. Show the result from your stdout { codec => rubydebug } output.

5

@magnusbaeck Thank you so much for your prompt reply. Here is an output with my current configuration:

{
        "message" => "167.220.232.218 - - [24/Apr/2017:15:39:30 +0900] \"GET /test/ HTTP/1.1\" 200 675 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2017-04-24T06:39:30.000Z",
           "path" => "/var/log/apache2/access.log",
           "host" => "yoichika-dev1",
           "type" => "apache_access",
       "clientip" => "167.220.232.218",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "24/Apr/2017:15:39:30 +0900",
           "verb" => "GET",
        "request" => "/test/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "675",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36\""
}
6

This is working perfectly fine. I don't know what you expected. If you wanted to convert the timestamp field in place rather than store the resulting date in @timestamp you need to use the date filter's target option.

1 Like
7

date filter's target is exactly what I wanted. I've tried with specifying custom name with target and was able to achieve what I wanted. Thanks alot!!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.