@timestamp not match with the log date

[Michael_TRAN]

Hi guys,

I'm new in the ELK universe. I have a trouble with the date of my log. On kibana, Timestamp shows the date when I put it on my server but not when the log were created. I try to modify my logstash configuration but it doesn't change.

Here the look of my log file

And the logstash configuration (etc/logstash/conf.d/12-apache-conf)

input {
file {
path => "/var/log/apache2/coc/*.log"
start_position => beginning
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
} else if [path] =~ "error" {
mutate { replace => { type => "apache_error" } }
}
date {
match => [ "mytimestamp", "yyyy/MM/dd:HH:mm:ss:SSSZ" ]
locale => "en"
add_tag => [ "tsmatch" ]
}
locale => "en"
}
}
output {

elasticsearch { host => localhost }

file { path => "/tmp/logstash.out"}
}

I see some topics on the forum but I don't find the solution.

Thanks for your help!

Regards !
Michael

[whyapenny]

your date match is wrong:
match => [ "mytimestamp", "yyyy/MM/dd:HH:mm:ss:SSSZ" ]

hard to tell from the screenshot, but looks like your date is actually:
dd/MMM/YYYY:HH:mm:ss: Z

Your joda template must match for date filter to match.

edit: modified my date match slightly

elastic/logstash/blob/v1.4.2/patterns/grok-patterns

[Michael_TRAN]

Hello,

Thank you for the reply. I do the change but it doesn't work.
Could you confirm that if I want to change the timestamp by the date inside message, I have to edit my 12-apache.conf in /etc/logstash/conf.d ?
And could I put 3 folder for analyse ? I put 3 path inside the 12-apache.conf.

Here my 12-apache.conf

input {
file {
path => "/var/log/apache2/cosm/*log"
path => "/var/log/apache2/cosg/*log"
path => "/var/log/apache2/coc/*log"
start_position => "beginning"
}
}

filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}

I restard Kibana, elasticsearch and logstash but it doesn't change anything.
Don't hesitate to click on the picture to have a better view =)

Thank you for your help
Michael

[whyapenny]

you should only need to restart logstash, not elasticsearch nor kibana. Can you please copy and paste one of your lines you are grokking. it will make it much easier to debug.

You can have multipile paths, that is fine. you can also have multiple date match filters such as:

match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z", "ISO8601", "dd/MM/YY:HH:mm:ss Z" ]

Also, i noticed i put lowercase Y's for the year match, i believe they should be upper case, so try that.

[Michael_TRAN]

Hello Whyapenny,

Sorry, I don't understand well this line ("Can you please copy and paste one of your lines you are grokking"). Are you looking for the example of line in the logs ? Or a specific line in the logstash configuration ?

188.20.113.34 - - [04/Mar/2017:23:47:48 +0100] "GET / HTTP/1.1" 200 4297 "-" "PATROL/V3.7.30i (Linux;INET KM 6.3.00 201001131830)"

Thanks for your help !

Michael

[guyboertje]

The YYYY should be lowercase.
I checked this in a date filter test that passes:

  describe "test test test" do
    config <<-CONFIG
      filter {
        date {
          match => [ "thedate", "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
      }
    CONFIG

    sample("thedate" => "04/Mar/2017:23:47:48 +0100") do
      expect(subject.get("@timestamp").to_s).to eq("2017-03-04T22:47:48.000Z")
    end
  end

[Michael_TRAN]

Hello guyboertje,

Thank you for your reply. I will try to put it in my logstash configuration file.

Regards,
Michael

[Michael_TRAN]

Hi guys,

I still have this issue. But the problem now, I don't see my data on kibana
Here the configuration of logstash (12-apache.conf)

filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

And the configuration of my filebeat.yml :

############################# Filebeat #####################################
filebeat:
  prospectors:
    -
      paths:
        - /var/log/apache2/coc/*.log
      fields:
      fields_under_root: true
      document_type: apache
  registry_file: /var/lib/filebeat/registry
############################# Output ##########################################
output:
  logstash:
    hosts: ["MyELKIP:5044"]
    tls:
      certificate_authorities: ['/etc/pki/tls/certs/logstash-forwarder.crt']      

Sometimes when I edit the logstash configuration file, I have some issue to start my filebeat service (SSL client failed to connect).
Do you have any idea about this ? Sorry, I'm beginning with this tool.

Regards,
Michael

[Michael_TRAN]

Hello,

I modify my logstash configuration, but on Kibana I still have the problem with the timestamp. It doesn't take the date of my log file. Here the configuration, I have see some topic talking about this problem but it doesn't working for me.

filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    remove_field => ["timestamp"]
    target => "@timestamp"
  }
}

Any idea about this ?
Thank you
Michael