Convert Array to String by ingesting three double quotes

Elastic Stack Logstash
1

Hi,

I am wanting to change an Array to a String to make it easier for me to store in my Elasticsearch, but I am having trouble finding the way of doing so. There are some special cases in my data where my field "value" will have an Array instead of a String, but I seem to not be able to add the three double quotes infront and behind my square brackets where I need them.

The following shows how my data looks like:

  {
    "Data": [{
            "key": "keyName",
            "value": "valueName"
        },
        {
            "key": "keyName",
            "value": "valueName"
        },
        {
            "key": "keyName",
            "value": [{
                "type": "FailedName",
                "message": "FailedMessage",
                "path": "FailedPath"
            }]
        }
    ]
  }

and here is how I want it to be inserted in ES:

  {
    "Data": [{
            "key": "keyName",
            "value": "valueName"
        },
        {
            "key": "keyName",
            "value": "valueName"
        },
        {
            "key": "keyName",
            "value": """[{
                "type": "FailedName",
                "message": "FailedMessage",
                "path": "FailedPath"
            }]"""
        }
    ]
  }

I have tried the mutate->gsub method but to no success.

Here is what my config file looks like

input {
	file{
		start_position => "beginning"
		path => "XXX.json"		
		sincedb_path => "/dev/null"
		codec => multiline {
			pattern => "^Spalanzani" 
			negate => true 
			what => previous 
			auto_flush_interval => 1 
			multiline_tag => "" 
		}
	}
	stdin{}
}

filter {
	mutate {
        gsub => [ "[data][customData][value]", "\[", '"""[', "[data][customData][value]", "\]", ']"""' ]
	}
	
	json {
	    source => "message"
	}
}

output {
	stdout {codec => rubydebug}
	elasticsearch {
		hosts => "hostAddress"
		user => "user"
		password => "password"
		index => "index" 
	}
}

Could anyone help me figure out how I could make it possible for me to insert three double quotes in front of my array to insert it as a string to ES?

2

To change the value from an array to a string you could use mutate+join

mutate {
    join => {
        "[Data][0][value]" => ","
        "[Data][1][value]" => ","
        "[Data][2][value]" => ","
        "[Data][3][value]" => ","
    }
}

or you could use a ruby filter. I have not tested it, but something like

ruby {
    code => '
        d = event.get("Data")
        if d.is_a? Array
            d.each_index { |x|
                if d[x]["value"].is_a? Array
                    d[x]["value"] = d[x]["value"].join(",")
                end
            }
            event.set("Data", d)
        end
    '
}
1 Like
3

Both of your solutions provided worked perfectly! Thank you Badger.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.