Converting to integer nested fields with ruby plugin

Elastic Stack Logstash
1

Hello.
I am attempting to convert to field type integer certain fields in my json.

{
"user_email": "example@com",
"user_id": "8ca5c5b4119b5d9c1b95d9e2af7e4702",
"path": "/tmp/mystuff.json",
"type": "json",
"statistics": {
"deleted_emails_number": "0",
"emails_number": "0",
"example.com": {
"template1": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
},
"template2": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
},
"example2.com": {
"template1": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
},
"template2": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
},
"exampleN.com": {
"templateN": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
}
}
}
}

I might have N templates and N domains, therefore I cannot hardcode it as I have tried before:
If I do not touch the nested fields I can do that with:

mutate {
convert => {
"[statistics][deleted_emails_number]" => "integer"
"[statistics][emails_number]" => "integer"
}
}

I might attempt to access nested fields like:

mutate {
convert => {
"[statistics][deleted_emails_number]" => "integer"
"[statistics][emails_number]" => "integer"
"[statistics][2][0][deleted_emails_number]" => "integer"
"[statistics][2][0][emails_number]" => "integer"
"[statistics][2][1][deleted_emails_number]" => "integer"
"[statistics][2][1][emails_number]" => "integer"
"[statistics][3][0][deleted_emails_number]" => "integer"
"[statistics][3][0][emails_number]" => "integer"
"[statistics][3][1][deleted_emails_number]" => "integer"
"[statistics][3][1][emails_number]" => "integer"
}
}
but that did not work as I expected.

How could I do it using ruby filter, but with nested fields?

ruby {
code => "
event.set('[statistics][deleted_emails_number]', event.get('[statistics][deleted_emails_number]').to_i)
"
}

Thank you for suggestions.

2

That's not going to work because none of the fields are arrays. You could do

mutate { convert => { "[statistics][example.com][exampleN.com][templateN][email_sent_count_number]" => "integer" } }

If you want to iterate over every field then you could do this

    ruby {
        code => '
            event.to_hash.each { |k1, v1|
                if k1 == "deleted_emails_number" or k1 == "email_sent_count_number"
                    event.set(k1, v1.to_i)
                elsif v1.is_a? Hash
                    v1.each { |k2, v2|
                        if k2 == "deleted_emails_number" or k2 == "email_sent_count_number"
                            event.set("[#{k1}][#{k2}]", v2.to_i)
                        elsif v2.is_a? Hash
                            v2.each { |k3, v3|
                                if k3 == "deleted_emails_number" or k3 == "email_sent_count_number"
                                    event.set("[#{k1}][#{k2}][#{k3}]", v3.to_i)
                                elsif v3.is_a? Hash
                                    v3.each { |k4, v4|
                                        if k4 == "deleted_emails_number" or k4 == "email_sent_count_number"
                                            event.set("[#{k1}][#{k2}][#{k3}][#{k4}]", v4.to_i)
                                        elsif v4.is_a? Hash
                                            v4.each { |k5, v5|
                                                if k5 == "deleted_emails_number" or k5 == "email_sent_count_number"
                                                    event.set("[#{k1}][#{k2}][#{k3}][#{k4}][#{k5}]", v5.to_i)
                                                end
                                            }
                                        end
                                    }
                                end
                            }
                        end
                    }
                end
            }
        '
    }

Obviously that needs to be refactored into something much cleaner.

3

Badger, that is very cryptic! :slight_smile:
I have to apology, I have made a mistake when doing my json above.
The manual mutate would look like:

mutate { 
convert => { "[statistics][example.com][template1][email_sent_count_number]" => "integer" } 
convert => { "[statistics][example.com][template1][deleted_emails_number]" => "integer" } 
convert => { "[statistics][example.com][template2][email_sent_count_number]" => "integer" } 
convert => { "[statistics][example.com][template2][deleted_emails_number]" => "integer" } 
convert => { "[statistics][example2.com][template1][email_sent_count_number]" => "integer" } 
convert => { "[statistics][example2.com][template1][deleted_emails_number]" => "integer" } 
convert => { "[statistics][example2.com][template2][email_sent_count_number]" => "integer" } 
convert => { "[statistics][example2.com][template2][deleted_emails_number]" => "integer" } 
}

Simplified nests below:

{
"statistics": {
"example.com": {
"template1": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
},
"template2": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
},
"example2.com": {
"template1": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
},
"template2": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
},
"exampleN.com": {
"templateN": {
"deleted_emails_number": "0",
"email_sent_count_number": "0"
}
}
}
}

Do I have to rewrite a great portion of your solution to adapt to this?

4

The script above would work for that JSON too, but it could be simplified...

    ruby {
        code => '
            event.to_hash.each { |k1, v1|
                if v1.is_a? Hash
                    v1.each { |k2, v2|
                        if v2.is_a? Hash
                            v2.each { |k3, v3|
                                if v3.is_a? Hash
                                    v3.each { |k4, v4|
                                        if k4 == "deleted_emails_number" or k4 == "email_sent_count_number"
                                            event.set("[#{k1}][#{k2}][#{k3}][#{k4}]", v4.to_i)
                                        end
                                    }
                                end
                            }
                        end
                    }
                end
            }
        '
    }
5

Thank you Badger for this contribution, I hope that the community will benefit from this answer.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.