Which ways to copy subfields of kubernetes.container._module.labels.* to kubernetes.lables.*?

Elastic Stack Logstash
1

Hi,

to simplify filtering I want to copy all substructure of kubernetes.container._module.labels.* to kubernetes.lables.*. Just moving a copy levels up. The structure below .labels is dynamic, so I cannot hardcode the labels.

What are the ways to go?
Number 1: writing a ruby script (I a not a big fan of ruby)
Number 2: Is there a way to copy structure just with mutate or so?
Number 3: your call :wink:

Thank you for your ideas.
Regards, Andreas

2

You can rename it with mutate

mutate { rename => { "[kubernetes][container][_module][labels]" => "[kubernetes][lables]" } }

You may then want to remove some fields.

3

thanks, I will try. I assume copy will work the same? I don't want to rename / move, because I don't want to break any shipped functionality like the infrastructure module.

4

I tried it out.

first try: copy [kubernetes][container][_module] to [kubernetes] was overwriting everything else what already was in structure kubernetes.

My successful approach now looks like this:

  # depending on the sub module of kubernetes, there are parameters like labels which are not always stored in kubernetes root object.
  # so logically identical data is stored in different objects - depending on the kubernetes resource.
  # Following filters will clone these objects to kubernetes root, so that kibana is able to filter all kubernetes resources by label.

  if [kubernetes][container][_module]
  {
		# labels are optional
		if [kubernetes][container][_module][labels]
		{
			mutate
			{
			  copy => { "[kubernetes][container][_module][labels]" => "[kubernetes][labels]" }
			}
		}

		# following fields are not optional
		mutate
    {
			copy => { "[kubernetes][container][_module][namespace]" => "[kubernetes][namespace]" }
			copy => { "[kubernetes][container][_module][node]" => "[kubernetes][node]" }
			copy => { "[kubernetes][container][_module][pod]" => "[kubernetes][pod]" }
    }
  }


  else if [kubernetes][pod][_module]
  {
		# labels are optional
		if [kubernetes][pod][_module][labels]
		{
			mutate
			{
				copy => { "[kubernetes][pod][_module][labels]" => "[kubernetes][labels]" }
			}
		}

		# following fields are not optional
	  mutate
    {
			copy => { "[kubernetes][pod][_module][namespace]" => "[kubernetes][namespace]" }
			copy => { "[kubernetes][pod][_module][node]" => "[kubernetes][node]" }
    }
  }


  else if [kubernetes][node][_module]
  {
		# labels are optional
		if [kubernetes][node][_module][labels]
		{
			mutate
	    {
	      copy => { "[kubernetes][node][_module][labels]" => "[kubernetes][labels]"}
	    }
		}
  }


  else if [kubernetes][volume][_module]
  {
		# labels are optional
		if [kubernetes][volume][_module][labels]
		{
		mutate
	    {
	      copy => { "[kubernetes][volume][_module][labels]" => "[kubernetes][labels]"}
	    }
		}

		# following fields are not optional
		mutate
		{
			copy => { "[kubernetes][volume][_module][namespace]" => "[kubernetes][namespace]" }
			copy => { "[kubernetes][volume][_module][node]" => "[kubernetes][node]" }
			copy => { "[kubernetes][volume][_module][pod]" => "[kubernetes][pod]" }
		}

  }


  else if [kubernetes][system][_module]
  {
		# labels are optional
		if [kubernetes][system][_module][labels]
		{
			mutate
	    {
	      copy => { "[kubernetes][system][_module][labels]" => "[kubernetes][labels]"}
	    }
		}

		# following fields are not optional
		mutate
		{
			copy => { "[kubernetes][system][_module][node]" => "[kubernetes][node]" }
		}
  }

Thanks for your help @Badger

5

mutate+copy is a no-op if a field does not exist, so you can compress this down to

mutate {
    copy => {
        "[kubernetes][container][_module][labels]"      => "[kubernetes][labels]"
        "[kubernetes][container][_module][namespace]"   => "[kubernetes][namespace]"
        "[kubernetes][container][_module][node]"        => "[kubernetes][node]"
        "[kubernetes][container][_module][pod]"         => "[kubernetes][pod]"
        "[kubernetes][pod][_module][labels]"            => "[kubernetes][labels]"
        "[kubernetes][pod][_module][namespace]"         => "[kubernetes][namespace]"
        "[kubernetes][pod][_module][node]"              => "[kubernetes][node]"
        "[kubernetes][node][_module][labels]"           => "[kubernetes][labels]"
        "[kubernetes][volume][_module][labels]"         => "[kubernetes][labels]"
        "[kubernetes][volume][_module][namespace]"      => "[kubernetes][namespace]"
        "[kubernetes][volume][_module][node]"           => "[kubernetes][node]"
        "[kubernetes][volume][_module][pod]"            => "[kubernetes][pod]"
        "[kubernetes][system][_module][labels]"         => "[kubernetes][labels]"
        "[kubernetes][system][_module][node]"           => "[kubernetes][node]"
    }
 }

assuming that only one of the sets of node/labels/pod will be present.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.