Hi,
I have problems converting my settings from logstash 2.x to logstash 5.2.
My log entry logs like this:
"@timestamp" => 2017-03-02T12:56:01.946Z,
"@version" => "1",
"index_date" => "2017-03-02",
"message" => "{\"meta\":{\"region\":\"staging\",\"tenantId\":\"424983274972394729348729834\"},\"creation_time\":1488459361,\"log\":{\"message\":\"Mar 2 12:55:34 proxy-server: 10.44.57.91 10.44.57.5 02\\/Mar\\/2017\\/12\\/55\\/34 GET \\/v1\\/AUTH_34-203490324-2394-23493342234%3Fformat%3Djson%26prefix%3Ddispersion_0 HTTP\\/1.0 200 - Python-urllib\\/2.7 gAAAAABYuBZGWXzx... - 2110 - txd908da9s0d80ad-3-0113-312 - 0.0887 - - 1488459334.769865990 1488459334.858609915 -\",\"dimensions\":{\"protocol\":\"HTTP\\/1.0\",\"ip\":\"10.44.57.142\",\"auth_token\":\"DDFSSDDESS...\",\"request_end_time\":\"1488459334.858609915\",\"request_start_time\":\"1488459334.769865990\",\"container\":\"%{container}\",\"request_time\":\"0.0887\",\"remote_addr\":\"10.44.57.5\",\"hostname\":\"test-server\",\"source\":\"-\",\"program\":\"proxy-server\",\"request_method\":\"GET\",\"type\":\"swift\",\"transaction_id\":\"tx9423-402394-023493-3423434\",\"client_ip\":\"10.2.3.1\",\"policy_index\":\"-\",\"object\":\"%{object}\",\"referer\":\"-\",\"log_info\":\"-\",\"path\":\"\\/var\\/log\\/swift\\/proxy.log\",\"response\":\"200\",\"request_path\":\"\\/v1\\/AUTH_a90da08d09ad80dads%3Fformat%3Djson%26prefix%3Ddispersion_0\",\"account\":\"2309420492309480324984\",\"bytes_recvd\":\"0\",\"bytes_sent\":\"2110\",\"headers\":\"-\",\"user_agent\":\"Python-urllib\\/2.7\",\"client_etag\":\"-\"}}}",
"tenant" => "%{[meta]}",
"tags" => [
[0] "_rubyexception"
In the past I used ruby filter to extract the fields:
if "dimensions" in [log] {
ruby {
code => "
fieldHash = event['log']['dimensions']
fieldHash.each do |key, value|
event[key] = value
end
"
}
}
to extract the dimensions.
I also used mutate to create new fields for tenant and region.
mutate {
add_field => {
region => "%{[meta][region]}"
}
}
Both are not working. Any idea, what I am doing wrong. Fideling around for days now and I have no clue.
Thanks a lot for your help and regards,
Olaf