Renaming nested fields

Elastic Stack Logstash
1

Hello,

I'm trying to rename the fields coming from a jdbc_streaming and has a target like [field1][nestedField],

in my document I get a list with the result of the query, since the jdbc_streaming filter plugin don't have the option "lowercase_column_names" i can't set up the names I want in the query and all comes back with lowercase.

So now I have this:

{
	"_source": {
		"field1": {
			"nestedField": [{
					"lowercase1": "value1",
					"lowercase2": "value2"
				},
				{
					"lowercase1": "value3",
					"lowercase2": "value4"
				}
			]
		}
	}
}

for my end result I want this:

{
	"_source": {
		"field1": {
			"nestedField": [{
					"camelCase1": "value1",
					"camelCase2": "value2"
				},
				{
					"camelCase1": "value3",
					"camelCase2": "value4"
				}
			]
		}
	}
}

i tried to use the mutate rename plugin with no success.
I used it like this:

mutate {
        rename => {
                "[field1][nestedField][lowercase1]" => "[field1][nestedField][camelCase1]"
                "[field1][nestedField][lowercase2]" => "[field1][nestedField][camelCase2]"
        }
}

What am I doing wrong?

Thanks in advance for any help!

1 Like
2

Your examples aren't valid JSON so it's hard to understand what your events actually look like.

3

Sorry @magnusbaeck, I messed up the curly braces.

I have this:


{
	"_source": {
		"field1": {
			"nestedField": [{
					"lowercase1": "value1",
					"lowercase2": "value2"
				},
				{
					"lowercase1": "value3",
					"lowercase2": "value4"
				}
			]
		}
	}
}

And I want this:

{
	"_source": {
		"field1": {
			"nestedField": [{
					"camelCase1": "value1",
					"camelCase2": "value2"
				},
				{
					"camelCase1": "value3",
					"camelCase2": "value4"
				}
			]
		}
	}
}

I tried to do it like this:

mutate {
        rename => {
                "[field1][nestedField][lowercase1]" => "[field1][nestedField][camelCase1]"
                "[field1][nestedField][lowercase2]" => "[field1][nestedField][camelCase2]"
        }
}
4

nestedField is an array, so you need to index into it.

"[field1][nestedField][0][lowercase1]" => "[field1][nestedField][0][camelCase1]"
1 Like
5

Hey @Badger, thanks for the anwer but that does not work for me, changes nothing, I still get the lowercase names

6

With this configuration

filter { json { source => "message" } }
input { generator { message => '{
        "field1": {
            "nestedField": [{
                    "lowercase1": "value1",
                    "lowercase2": "value2"
                },
                {
                    "lowercase1": "value3",
                    "lowercase2": "value4"
                }
            ]
        }
    }' count => 1 } }
output { stdout { codec => rubydebug } }
filter {
    mutate {
        rename => {
            "[field1][nestedField][0][lowercase1]" => "[field1][nestedField][0][camelCase1]"
            "[field1][nestedField][0][lowercase2]" => "[field1][nestedField][0][camelCase2]"
        }
    }
}

I get

        "field1" => {
        "nestedField" => [
            [0] {
                "camelCase1" => "value1",
                "camelCase2" => "value2"
            },
            [1] {
                "lowercase2" => "value4",
                "lowercase1" => "value3"
            }
        ]
    },

What does stdout { codec => rubydebug } produce for your messages?

7

Hey @Badger,

the thing is i don't want to change only the first element, I want to change the name of all elements inside the array.

your solution only changes the first element.

8

The mutate+rename example you gave in the first post only renamed the first array entry. I showed you how to fix that example.

If that's not the problem you want to solve please explain what you do want to solve.

9

The mutate-rename example in my first post changes nothing, but I think I was clear with the examples I provided, what I have vs want I want.

10

I feel I'm getting closer,

I've found on another post a way to do it but somehow it's not working for me.

I used the filter ruby plugin like this

filter {
	if([field1][nestedField]){
		ruby {
			code => "
				event.get('[field1][nestedField]').each { |k|
					k['camelCase1'] = k['lowercase1']
					k['camelCase2'] = k['lowercase2']
					k.delete('lowercase1')
					k.delete('lowercase2')
					logger.info('for each k', 'value' => k)
				}
				logger.info('full array' , 'value' => event.get('[field1][nestedField]'))
			"
		}
	}
}

The thing is when I print "k" in the logger it's exactly has I want it to be but when I print the whole array nothing has changed.

Am I missing some kind of set?

11

Try this

        code => "
            b = []
            event.get('[field1][nestedField]').each { |k|
                k['camelCase1'] = k['lowercase1']
                k['camelCase2'] = k['lowercase2']
                k.delete('lowercase1')
                k.delete('lowercase2')
                logger.info('for each k', 'value' => k)
                b << k
            }
            event.set('[field1][nestedField]', b)
        "
6 Likes
12

Yes @Badger that did the trick!

Thank you for your help.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.