I cannot get logstash to format custom field to be suitable for sort in kibana.
pipeline filter config is:
filter {
if ([fields][log_type] == "zira_prod_log") {
grok {
match => { "message" => "[%{TIMESTAMP_ISO8601:timestamp}] %{NOTSPACE:log_type}: %{NOTSPACE:field3} [%{NOTSPACE:verb}] %{NOTSPACE:api_uri} %{GREEDYDATA:jsonstring1} %{GREEDYDATA:jsonstring2}" }
patterns_dir => ["/etc/logstash/.patterns"]
}
mutate {
rename => { "source" => "file" }
replace => { "[type]" => "%{[fields][environment]}-zira_prod_log" }
}
date {
timezone => "Europe/Zagreb"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss.SSS", "ISO8601" ]
target => "@timestamp"
}
}
}
neither @timestamp is overwritten with timestamp nor is timestamp sortable (because it gets type text and not date)
input is beats, output elastic, i tried multiple variations, always delete es index, restart logstash on pipeline changes, and recreate index pattern, but no combo i've tried helped
how es shows field:
"type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }
log example:
[2020-03-11 14:38:18.123456] zira_api.INFO: REQUEST [GET] https://x.y.z/examplepath/category?filter... {"payload":"[object] (Infrastructure\External\...: )"} {"tags":{"request_id":"d8886a16-..."}}