Logstash configuration | Elimination of if loops in checking values at nested JSON keys

Elastic Stack Logstash
#1

Hello Community Users,

I am seeking help regarding writing a better logstash configuration file.

I am dealing with nested array JSON which looks something like this.

{
  "data": {
    "content": {
      "payload": [
        {
          "a": "John",
          "b": "Doe",
          "c": 23
        },
        {
          "b": "Mary",
          "c": "Smith"
        }
      ]
    }
  }
}

And to parse this JSON, I am writing if conditions in the JSON filter I am using in my logstash configuration file.

filter{

	if[data][content][payload][0]{
	
		mutate {
		
			add_field => {
			
				"dataContext0" => '%{[data][content][payload]}%'
			}
		}
	}
	
	
	
	if[data][content][payload][1]{
	
		mutate {
		
			add_field => {
			
				"dataContext1" => '%{[data][content][payload]}%'
			}
		}
	}
}

This code snippet works absolutely fine. The only problem is, I might have a thousand nested fields in the future in the JSON, and parsing like this would create a very ugly-looking configuration file as there will be a thousand number of if conditions I will have to add before adding them in mutating fields.

I wanted to know, Is there any better way to deal with this which can reduce these if conditions or eliminate them at all with a better alternative.

Please help me here if you know something or have gone through a scenario like this.

Thanks in advance.

#2

You could do that using a ruby filter. I have not tested this, but something like

ruby {
    code => '
        payload = event.get("[data][content][payload]")
        if payload
            payload.each_index { |x|
                event.set("dataContext#{x}", payload[x])
            }
        end
    '
}
#3

Thank you @Badger for a prompt reply.

I tired the solution given by you. However, using that scripting gave me an error something like this.

[ERROR][logstash.filters.ruby ][main][2995e4e436ef916873906373f353bb6bb6eff43deb775d330c06d066d97bc3d9] Ruby exception occurred: undefined method each_index' for #Hash:0x21fb0c31`

So, I looked for changed "each_index" function with "each" as per solutions provided in other threads. Like this :

filter{	
	json{
		source => "message"
		target => "new"
	}
	split{
		field => "[new][Records]"
	}
	mutate{
		add_field => {
			"content" 	=> '%{[new][Records][Data][payload][content]}'
		}
	}	
	json{
		source => "content"
		target => "contents"
	}

	ruby {
	    code => '
	        contents = event.get("[contents]")
	        if contents
	            contents.each { |x|
	                event.set("{x}", contents[x])
	            }
	        end
	    '
	}

	mutate{
		remove_field => ["path", "@version", "host", "tags", "fields", "message","new","contents","eventType","taskDefinition","priority","content"]
	}
}

But this solution is throwing an error for a field which is already configured before scripting ruby.

[2021-02-23T22:56:12,674][ERROR][logstash.filters.ruby    ][main][c1ecdd2dc6b4374ad48846533a8813f0fad9b1a694695b8bcafdb43287264129] Ruby exception occurred: undefined local variable or method `payload' for #<LogStash::Filters::Ruby:0x58283990>
[2021-02-23T22:56:12,722][ERROR][logstash.filters.ruby    ][main][c1ecdd2dc6b4374ad48846533a8813f0fad9b1a694695b8bcafdb43287264129] Ruby exception occurred: undefined local variable or method `payload' for #<LogStash::Filters::Ruby:0x58283990>
[2021-02-23T22:56:12,885][ERROR][logstash.filters.ruby    ][main][c1ecdd2dc6b4374ad48846533a8813f0fad9b1a694695b8bcafdb43287264129] Ruby exception occurred: undefined local variable or method `payload' for #<LogStash::Filters::Ruby:0x58283990>
[2021-02-23T22:56:12,941][ERROR][logstash.filters.ruby    ][main][c1ecdd2dc6b4374ad48846533a8813f0fad9b1a694695b8bcafdb43287264129] Ruby exception occurred: undefined local variable or method `payload' for #<LogStash::Filters::Ruby:0x58283990>
[2021-02-23T22:56:12,970][ERROR][logstash.filters.ruby    ][main][c1ecdd2dc6b4374ad48846533a8813f0fad9b1a694695b8bcafdb43287264129] Ruby exception occurred: undefined local variable or method `payload' for #<LogStash::Filters::Ruby:0x58283990>

Can you please guide me here ?

#4

That is telling you that [data][content][payload] is a Hash, not an Array. It may be that when there is only one Array member it just presents a Hash instead of an Array of Hash, in which case you could change the ruby file to us

if payload.is_a? Array

instead of

if payload

For you other question I cannot answer without seeing what the data looks like.

#5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.