Ruby filter to pack string value into object

Elastic Stack Logstash
1

I have a client that sends HTTP request events with a nested structure, like:

context.response.body
context.response.code
context.response.headers.Content-Length
context.response.headers.Content-Type
etc..

But sometimes the context.response field is sent as a string, which is a problem for Elasticsearch.
I want to make a Ruby filter to fix this, so if it's a string, pack it into context.response.body instead.

This is what I came up with initially (having never written any Ruby before):

if event.include? '[context][response]' && event.get('[context][response]').is_a? String
        textval = event.get('[context][response]')
        event.remove('[context][response]')
        event.set('[context][response][body]', textval)
end

Logstash just crashes immediately, with the error:

SyntaxError: (ruby filter code):3: syntax error, unexpected tCONSTANT                   
if event.include? '[context][response]' && event.get('[context][response]').is_a? String
^

Any help is greatly appreciated :slight_smile:

2

Hi,

If you have any field type issues i'd recommend to use the convert plugin in logstash ? Mutate filter plugin | Logstash Reference [8.11] | Elastic

does this fit your use case ?

3

Yes, that actually fits it perfectly!

I didn't realize it would just "do nothing" if run on a hash. Thank you!

EDIT: Sorry, was too quick.
The convert process can't turn a string into a hash, as far as I can see.
The issue is I need to check:

if context.response is type String:
  move context.response into context.response.body

Moving it into a different field would also be acceptable, but most of the events contain context.response as a hash, so those need to be left untouched.

1 Like
4

When context.response is an object is there any nested field that is always present? Like will you always have context.response.body or context.response.code?

If so, than you can fix this issue without the need to write any ruby, just combining some mutate filters.

5

Yes, it will always have those fields when it's an object.
Which mutate filters? How can I detect when it's an object/string?

Edit: Also still interested in why the Ruby code is failing

6

That looks a lot like this.

7

Not sure why, but the new code works:

if event.include? '[context][response]' and event.get('[context][response]').is_a? String
  event.set('[context][response][body]', event.remove('[context][response]'))
end

Thank you!

8

If you always have a specific field when the field is an object you can text if this field exists, if it does not exist, than you do not have it as an object.

For example, using this sample events:

{ "context": { "response": "response_is_string"}}
{ "context": { "response": {"body": "response_is_hash"}}}

The following filter will work:

filter {
    json {
        source => "message"
    }
    if ![context][response][body] {
        mutate {
            rename => {
                "[context][response]" => "[context][response][body]"
            }
        }
    }

}

It tests if the field context.response.body exists, if it does not exists than it will rename context.response to context.response.body, but this only works if you can guarantee that when context.response is an object, it will always have the nested field body.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.