Replacing all instances of a string in nested JSON

[7armyant]

Hi Gurus,

I'm new to logstash and have had little success with the following.
I have been trying to relace all instances of a sting in nested JSON.
Below is the JSON in question:

"{"totalCount":
	2,
"nextPageKey":
	null,
"resolution":
	"Inf",
"result":[
	{"metricId":
		"builtin:synthetic.browser.availability.location.totalWoMaintenanceWindow",
	"data":[
		{"dimensions":
			["SYNTHETIC_TEST-XXXXXXXXXXXX","GEOLOCATION-YYYYYYYYYYYYY"],
		"dimensionMap":
			{"dt.entity.synthetic_test":
				"SYNTHETIC_TEST-XXXXXXXXXXXX",
			"dt.entity.geolocation\":
				"GEOLOCATION-YYYYYYYYYYYYY"},
		"timestamps":
			[1646705640000],
		"values":
			[100]
		},

		{"dimensions":[
			"SYNTHETIC_TEST-XXXXXXXXXXXX","GEOLOCATION-XXXXXXXXXXXX"],
		"dimensionMap":
			{"dt.entity.synthetic_test":
				"SYNTHETIC_TEST-XXXXXXXXXXXX",
			"dt.entity.geolocation":
				"GEOLOCATION-XXXXXXXXXXXX"},
		"timestamps":
			[1646705640000],
		"values":
			[100]
	}
	]}
]}"

The values I need to replace are the SYNTHETIC_TEST-XXXXXXXXXXXX and GEOLOCATION-XXXXXXXXXXXX values with a variable in the logstash config.

I then need to pull out each of the "values" and create a field like such:
{GEOVAR1} => 100
{GEOVAR2} => 100

Any help would be greatly appreciated.

Thanks

[Badger]

@7armyant

The values I need to replace are the SYNTHETIC_TEST-XXXXXXXXXXXX and GEOLOCATION-XXXXXXXXXXXX values with a variable in the logstash config.

What do you mean by "variable"? Is it the value of another field?

[7armyant]

Hi Badger,

I have recently taken over the administration of our logstash instances.

We currently have an environment file that is loaded on the start of the logstash instance.
That file contains a number of variables that are used in various pipelines for things such as environments or authentication tokens.
I was hoping to populate that file with variables for the GEOLOCATIONS to save time building and maintaining the many pipelines ill be building as part of this.

If this is not recommended then specifying them in the pipelines is also fine.

[Badger]

Using environment variables is fine. You can reference them in the configuration.

If there are always two entries in the array you could do it using

    json { source => "message" remove_field => [ "message" ] }
    mutate {
        gsub => [
            "[result][0][data][0][dimensions][0]", "SYNTHETIC_TEST-\w+", "${SYNTHETIC_TEST}",
            "[result][0][data][0][dimensions][1]", "GEOLOCATION-\w+", "${GEOLOCATION}",
            "[result][0][data][0][dimensionMap][dt.entity.synthetic_test]", "SYNTHETIC_TEST-\w+", "${SYNTHETIC_TEST}",
            "[result][0][data][0][dimensionMap][dt.entity.geolocation]", "GEOLOCATION-\w+", "${GEOLOCATION}",

            "[result][0][data][1][dimensions][0]", "SYNTHETIC_TEST-\w+", "${SYNTHETIC_TEST}",
            "[result][0][data][1][dimensions][1]", "GEOLOCATION-\w+", "${GEOLOCATION}",
            "[result][0][data][1][dimensionMap][dt.entity.synthetic_test]", "SYNTHETIC_TEST-\w+", "${SYNTHETIC_TEST}",
            "[result][0][data][1][dimensionMap][dt.entity.geolocation]", "GEOLOCATION-\w+", "${GEOLOCATION}"
        ]
    }
    mutate {
        add_field => {
            "GEOVAR1" => "%{[result][0][data][0][values][0]}"
            "GEOVAR2" => "%{[result][0][data][1][values][0]}"
        }
    }

If the array ever changes size then I would iterate over it in ruby...

    json { source => "message" remove_field => [ "message" ] }
    ruby {
        code => '
            result = event.get("result")
            if result.is_a? Array
                data = result[0]["data"]
                if data.is_a? Array
                    data.each_index { |i|
                        event.set("GEOVAR#{i+1}", data[i]["values"][0])

                        data[i]["dimensions"][0] =
                            data[i]["dimensions"][0].sub(/SYNTHETIC_TEST-\w+/, ENV["SYNTHETIC_TEST"])
                        data[i]["dimensionMap"]["dt.entity.synthetic_test"] =
                            data[i]["dimensionMap"]["dt.entity.synthetic_test"].sub(/SYNTHETIC_TEST-\w+/, ENV["SYNTHETIC_TEST"])

                        data[i]["dimensions"][1] =
                            data[i]["dimensions"][1].sub(/GEOLOCATION-\w+/, ENV["GEOLOCATION"])
                        data[i]["dimensionMap"]["dt.entity.geolocation"] =
                            data[i]["dimensionMap"]["dt.entity.geolocation"].sub(/GEOLOCATION-\w+/, ENV["GEOLOCATION"])
                    }
                    event.set("[result][0][data]", data)
                end
            end
        '
    }

[7armyant]

You are a man of magic.
That worked for me perfectly.

Thank you so much.