Translate Config works from STDIN but not Beats input

Elastic Stack Logstash
1

The following code works if I paste the data into STDIN, but when the data comes from BEATS it fails to create the geoip fields. I just get a translation field with the result in it.

#This is from the logstash config

	#Private IP Geo Lookup
	if "_geoip_lookup_failure" in [tags] {
		translate {
			regex => true
			dictionary_path => "./config/priv_ip_trans.yml"
			field => "CallingStation_IP"
			remove_tag =>"_geoip_lookup_failure"
		}
	}

#This is a sample line from the priv_ip_trans.yml
'10.90.3.*': '{"geoip":{"Latitude": 37.7576948, "Longitude": -122.4726193, "Location": [-122.4726193, 37.7576948],"country_name": "United States", "country_code2": "US", "country_code3": "US", "continent_code":" NA", "city_name": "San Francisco", "region_code": "CA", "region_name": "California", "postal_code": "94114", "timezone": "America/Los_Angeles"}}'

#This is what gets output when using beats:

              "translation" => "{\"geoip\":{\"Latitude\": 37.7576948, \"Longitude\": -122.4726193, \"Location\": [-122.4726193, 37.7576948],\"country_name\": \"United States\", \"country_code2\": \"US\", \"country_code3\": \"US\", \"continent_code\":\" NA\", \"city_name\": \"San Francisco\", \"region_code\": \"CA\", \"region_name\": \"California\", \"postal_code\": \"94114\", \"timezone\": \"America/Los_Angeles\"}}",

This is what I get when using STDIN:

      "geoip" => {
         "city_name" => "San Francisco",
          "timezone" => "America/Los_Angeles",
      "country_name" => "United States",
     "country_code2" => "US",
     "country_code3" => "US",
    "continent_code" => " CA",
          "Latitude" => 36.0768949,
       "region_name" => "California",
       "postal_code" => "94114",
         "Longitude" => -122.4726193,
          "Location" => [
        [0] 36.0768949,
        [1] -122.4726193
    ],
       "region_code" => "CA"
},
2

To me it sounds very unlikely that the translate filter would behave differently depending on the input. You can always use a json filter to deserialize the JSON string.

Your stdin example doesn't match your sample line from priv_ip_trans.yml. Had you shown the entire event from your stdout output it would've been easier to see if that was expected.

3

You were right, I was missing the JSON Filter. Don't know how I missed that. Thanks for the kick in the head to straighten my thoughts out.