JSON formatted log contains "message" field confusing Logstash

Elastic Stack Logstash
11

Ok, I've gotten some time to work on this and the craziness continues. Still struggling with (I think) keeping everything unique. So for most of my logs I have the following (were host_type is set by the logstash-forwarder with the name of the Tomcat application the host is running).

if [type] == "log4j-json" {
 json	{
  source => "message"
  target => "log_data"	
  }
 
 date {
   locale => en
   match => [ "[log_data][timestamp]" , "yyyy-MM-dd HH:mm:ss,SSS" , "ISO8601" ]
  }

mutate {
  remove_field => [ "message" ]
  rename => { "log_data" => "%{host_type}" }
  }
}

The incoming files are JSON formatted. Sort of (valid JSON but they are ugly inside) for example for the host_type "rm-notification-ws-dev-dev" (don't ask):

{"message":"action=process_notification_messages , payload={\"id\":\"AVnwGgYUCMuYDvqxa8YD\",\"deliveryType\":\"PUSH-NOTIFICATION\",\"notificationType\":\"EVENT-SUBSCRIPTION\",\"userId\":\"gus@ail.com\",\"userHash\":\"e5f358e424bf321f8ac5ae786d2907be\",\"notificationTimestamp\":\"2017-01-30T15:59:14.441Z\",\"payload\":{\"operation\":\"probe-sample-event\",\"siteDetail\":{\"siteId\":108465,\"address1\":\"125 POWDER FOREST DRIVE\",\"siteName\":\"TLechak Office TLS-450P\"},\"deviceDetail\":{\"deviceType\":\"TLS350\",\"deviceTypeDescription\":\"TLS350\",\"deviceId\":\"1244922\"}},\"notificationCategory\":\"info\"}","timestamp":"2017-01-30 15:59:14,950","loggerName":"sys.consumer","level":"INFO","threadName":"ActiveMQ Session Task-11898"}

This works as expected (hurray).

But other applications, with very similar formatting fail (host_type:

{"message":{"description":"Package not found","exception":"com.gilbarco.i360.core.error.ApplicationException: TASK_NOT_FOUND: TASK_NOT_FOUND - null","user_agent":"Jodd","code":"TASK_NOT_FOUND","type":"checked","url":"/passport-core/action/scheduled-task-v1/schedule-callback"},"timestamp":"2017-01-30 04:29:00,180","loggerName":"sys.web","level":"WARN","threadName":"2417383241"}

I cannot figure out what is going on here. It never accepts this line - there is no mapping for the second host_type's messages - it's not taking any of them! Here is what logstash says:

{:timestamp=>"2017-01-30T16:06:30.309000+0000", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-dev-2017.01.30", :_type=>"log4j-json", :_routing=>nil}, #<LogStash::Event:0x6efd0f07 @metadata_accessors=#<LogStash::Util::Accessors:0x4f52bb36 @store={}, @lut={}>, @cancelled=false, @data={"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", "exception"=>"com.gilbarco.i360.core.error.ApplicationException: TASK_NOT_FOUND: TASK_NOT_FOUND - null", "user_agent"=>"Jodd HTTP", "code"=>"TASK_NOT_FOUND", "type"=>"checked", "url"=>"/passport-core/action/scheduled-task-v1/schedule-callback"}, "timestamp"=>"2017-01-30 04:29:00,180", "loggerName"=>"sys.web", "level"=>"WARN", "threadName"=>"2417383241"}}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x583a16e6 @store={"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", <snip>}, @lut={"i360-core-dev"=>[{"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", "exception"=>"com.gilbarco.i360.core.error.ApplicationException: TASK_NOT_FOUND: TASK_NOT_FOUND - null", "user_agent"=>"Jodd HTTP", "code"=>"TASK_NOT_FOUND", "type"=>"checked", "url"=>"/passport-core/action/scheduled-task-v1/schedule-callback"}, "timestamp"=>"2017-01-30 04:29:00,180", "loggerName"=>"sys.web", "level"=>"WARN", "threadName"=>"2417383241"}}, "i360-core-dev"], "message"=>[{"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", "exception"=>"ore.error.ApplicationException: TASK_NOT_FOUND: TASK_NOT_FOUND - null", "user_agent"=>"Jodd HTTP", "code"=>"TASK_NOT_FOUND", "type"=>"checked", "url"=>"/passport-core/action/scheduled-task-v1/schedule-callback"}, "timestamp"=>"2017-01-30 04:29:00,180", "loggerName"=>"sys.web", "level"=>"WARN", "threadName"=>"2417383241"}}, "message"], **"[type]"=>**[{"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", "loggerName"=>"sys.web", "level"=>"WARN", "threadName"=>"2417383241"}}, "type"], "type"=>[{"@version"=>"1", "@timestamp"=>"2017-01-30T04:29:00.180Z", "file"=>"/var/log/tomcat7/gilbarco-passport-core.json", "host"=>"i360-core-ip-10-4-29-236", "host_type"=>"i360-core-dev", "type"=>"log4j-json", "i360-core-dev"=>{"message"=>{"description"=>"Package not found", "loggerName"=>"sys.web", "level"=>"WARN", "threadName"=>"2417383241"}}, "type"]}>>], :response=>{"create"=>{"_index"=>"logstash-dev-2017.01.30", "_type"=>"log4j-json", "_id"=>"AVnwIKrYH7RsVvf3n9Al", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [i360-core-dev.message]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"unknown property [description]"}}}}, :level=>:warn}

I can't look at the mappings because logstash has not accepted any messages to generate them with!