Trouble Ingesting JSON

[wwalker (Walker)]

I am trying to ingest a JSON document. Unfortunately, there seems to be a problem with null values on an object field. There are multiple fields like this, how can I resolve it?

When the field is null, it looks like this:

	"assigned_to": "",

However, when there is content, the field looks like this:

	"assigned_to": {
		"link": "https://ba",
		"value": "ba"
	},

I get an error in logstashing stating:

"failed to parse field [assigned_to] of type [text] in document with id '59k36GsBxmyDjcqePz6P'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:2174"}}}}}

Pipeline

input {
  file {
    id => "Ingest"
    path => "/*.json"
    stat_interval => 2
    start_position => "beginning"
    close_older => 300
  }
}
filter {
  json {
    id => "Parse"
    source => "message"
  }
}
output {
  elasticsearch {
    id => "Send to Elasticsearch"
  }
}

[Badger]

That's expected. In elasticsearch can be an object, or text (or date etc.), but it cannot be text on some documents and an object on others.

    if [assigned_to] == "" {
        mutate { rename => { "[assigned_to]" => "[assigned_to][value]" } }
    }

[wwalker (Walker)]

Excellent, is there a way to apply this kind of logic to all fields? The data I am ingesting has about 100 fields and, while I am uncertain as to how many will be like this, I'd prefer to not have to create a long, monotonous pipeline for it.

[Badger]

@wwalker

Excellent, is there a way to apply this kind of logic to all fields?

You might be able to do it using a ruby filter and event.to_hash.each.