Timestamp stdout is correct but to es is not

Elastic Stack Logstash
1

I have a log file which is customized by myself, I want to use timestamp by my log and write config as following shows:

input {
file {
    path => "/var/log/testwrd.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
}
}
filter {
grok {
    match => {
        "message" => [
            "\[%{TIMESTAMP_ISO8601:time}\] API %{WORD:type} IP:\[%{IP:ip}\] MAC:\[%{MAC:mac}\] DEVICE:\[%{DATA:device_type}-%{DATA:device_sub_type}\] SERVER:\[%{IP:from_ip}\] \[ %{WORD:action} \]",
            "\[%{TIMESTAMP_ISO8601:time}\] API %{WORD:type} IP:\[%{IP:ip}\] MAC:\[%{MAC:mac}\] DEVICE:\[%{DATA:device_type}-%{DATA:device_sub_type}\] SERVER:\[%{IP:from_ip}\] DEST:\[%{URI:dest}\] ERROR: \[%{DATA:error}\] INFO: \[%{DATA:info}\] RETURNCODE:\[%{NUMBER:rtn_code}\]",
            "\[%{TIMESTAMP_ISO8601:time}\] API %{WORD:type} IP:\[%{IP:ip}\] MAC:\[%{MAC:mac}\] DEVICE:\[%{DATA:device_type}-%{DATA:device_sub_type}\] SERVER:\[%{IP:from_ip}\] DEST:\[%{URI:dest}\] RETURNCODE:\[%{NUMBER:rtn_code}\]",
            "\[%{TIMESTAMP_ISO8601:time}\] API %{WORD:type} IP:\[%{IP:ip}\] MAC:\[%{MAC:mac}\] SERVER:\[%{IP:from_ip}\] DEST:\[%{URI:dest}\] RETURNCODE:\[%{NUMBER:rtn_code}\]",
            "\[%{TIMESTAMP_ISO8601:time}\] API %{WORD:type} IP:\[%{IP:ip}\] MAC:\[%{MAC:mac}\] SERVER:\[%{IP:from_ip}\] DEST:\[%{URI:dest}\] \[ %{WORD:action} \]"
        ]
    }
}
if [action] != "FILTERED" {
    mutate {
        add_field => {
            "action" => "PUSH"
        }
    }
}
date {
    match => ["time", "yyyy-MM-dd HH:mm:ss"]
    target => "@timestamp"
}
mutate {
    remove_field => ["time"]
}
}
output {
    stdout { codec => rubydebug  }
    #elasticsearch {
    #    hosts => ["10.123.11.95:9200", "10.123.11.180:9200"]
    #    index => "dhcp_notice-2017.10.23"
    #}
}

here is a example input:
In console, it can be parsed correctly:

image
image.png1877×737 58.5 KB

but to elasticsearch, in kibana, I see:

image
image.png1016×374 18.5 KB

why two different result? Thanks.

Well I thought maybe ISO8601 not compatible with my date format, so I changed it to DATA, but still no use...

2

Hi @imqishi,

did you try the grok debugger? there you can test if the pattern works for your timestamp also your grok pattern.

Grok debugger

Also the event in logstash and the event in Kibana are not the same :confused:

Cheers,
Dirk

3

Oh...I lost a pattern for these logs so some logs timestamp became now ...really stupid....
Thanks a lot!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.