Logstash 5.x ruby hash and mutate filter are crashing or not working

Hi,

I have problems converting my settings from logstash 2.x to logstash 5.2.

My log entry logs like this:

"@timestamp" => 2017-03-02T12:56:01.946Z,
  "@version" => "1",
"index_date" => "2017-03-02",
   "message" => "{\"meta\":{\"region\":\"staging\",\"tenantId\":\"424983274972394729348729834\"},\"creation_time\":1488459361,\"log\":{\"message\":\"Mar  2 12:55:34 proxy-server: 10.44.57.91 10.44.57.5 02\\/Mar\\/2017\\/12\\/55\\/34 GET \\/v1\\/AUTH_34-203490324-2394-23493342234%3Fformat%3Djson%26prefix%3Ddispersion_0 HTTP\\/1.0 200 - Python-urllib\\/2.7 gAAAAABYuBZGWXzx... - 2110 - txd908da9s0d80ad-3-0113-312 - 0.0887 - - 1488459334.769865990 1488459334.858609915 -\",\"dimensions\":{\"protocol\":\"HTTP\\/1.0\",\"ip\":\"10.44.57.142\",\"auth_token\":\"DDFSSDDESS...\",\"request_end_time\":\"1488459334.858609915\",\"request_start_time\":\"1488459334.769865990\",\"container\":\"%{container}\",\"request_time\":\"0.0887\",\"remote_addr\":\"10.44.57.5\",\"hostname\":\"test-server\",\"source\":\"-\",\"program\":\"proxy-server\",\"request_method\":\"GET\",\"type\":\"swift\",\"transaction_id\":\"tx9423-402394-023493-3423434\",\"client_ip\":\"10.2.3.1\",\"policy_index\":\"-\",\"object\":\"%{object}\",\"referer\":\"-\",\"log_info\":\"-\",\"path\":\"\\/var\\/log\\/swift\\/proxy.log\",\"response\":\"200\",\"request_path\":\"\\/v1\\/AUTH_a90da08d09ad80dads%3Fformat%3Djson%26prefix%3Ddispersion_0\",\"account\":\"2309420492309480324984\",\"bytes_recvd\":\"0\",\"bytes_sent\":\"2110\",\"headers\":\"-\",\"user_agent\":\"Python-urllib\\/2.7\",\"client_etag\":\"-\"}}}",
    "tenant" => "%{[meta]}",
      "tags" => [
    [0] "_rubyexception"

In the past I used ruby filter to extract the fields:

if "dimensions" in [log] {
    ruby {
        code => "
            fieldHash = event['log']['dimensions']
            fieldHash.each do |key, value|
                event[key] = value
            end
        "
    }
}

to extract the dimensions.
I also used mutate to create new fields for tenant and region.

mutate {
    add_field => { 
      region => "%{[meta][region]}"
    }
}

Both are not working. Any idea, what I am doing wrong. Fideling around for days now and I have no clue.

Thanks a lot for your help and regards,

Olaf

The event API changed on Logstash 5.x, it now uses getter/setter methods - https://www.elastic.co/guide/en/logstash/current/event-api.html

So your ruby code should be something like

if "dimensions" in [log] { ruby { code =" fieldHash = event.get('[log][dimensions]') fieldHash.each do |key, value| event.set(key, value) end " } }

Hi,

it is not working out of the box with that filter. I first have to add the json parser otherwise nothing is happening.

json {
  source => "message"
}

Any idea why? In the past I didn't have to do that.

Regards,

Olaf

It seems to me that it should be required to use a JSON codec even in 2.x version.
What kind of inputs do you have?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.