Hello,
I need assistance in creating an index pattern so I can search logs in Kibana. I have the following logstash configuration:
input {
beats {
port => 5044
host => "0.0.0.0"
}
}
filter {
json {
source => "message"
}
mutate {
convert => {
"startTime" => string
}
}
date {
match => [ "startTime" , "yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'" ]
timezone => "UTC"
target => "@timestamp"
}
mutate {
remove_field => [ "startTime", "@version", "tags", "message", "ecs", "agent", "input", "host" ]
}
}
output {
elasticsearch {
hosts => "${es_host}"
user => "${es_user}"
password => "${es_pwd}"
index => "xxx-development-%{+YYYY.MM.dd}"
ilm_enabled => true
ilm_rollover_alias => "xxx-development"
ilm_policy => "xxx-development"
}
}
I have the following date field defined in the index template:
"properties": {
"@timestamp": {
"type": "date"
},
An example log message read from filebeat before sent to Logstash looks like this:
{"startTime":"2021-12-02T05:56:04.696Z","level":"FATAL","serviceName":"ABC","pid":3674,"logId":"App Unhandled Rejection","data":"blah" ,"ServicePid":3674}}}
Essentially the startTime field is being sent as @timestamp as per above logstash config. I have previously created index patterns before for other templates using the same method but now it does not work.
When I got to Kibana->Index Patterns->Create index pattern, I can see matching sources for xxx-development-*, however on step 2 it picks up no time fields.
What can I do?
