# Why different behavior on @metadata vs normal fields?

**URL:** https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028
**Category:** Logstash
**Created:** [November 15, 2019, 9:51am UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028 "2019-11-15T09:51:47Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)
#### Post date: [November 15, 2019, 9:51am UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028/1 "2019-11-15T09:51:47Z")

</div>

Hi,

I have json data which is harvested by filebeat and shipped to logstash.  
Filebeat encapsulating the orginal json message in message field by default. I didn't change it.

What I want to do (in logstash):

- parse json string stored in message
- for the case that parsing fails, I want to keep the original input message for debugging purpose.
- if parsing succeeds, the original message should be dropped.
- it is possible that the original application log json contains a field named message. This must effect the message field created by filebeat.

My implementation idea looks like the following:

```
filter
{

	# remember logType
	mutate
	{

		# If included json may contain a field also called message, an array would be created
		# We don't want arrays, that's why we rename it.
		rename => ["message", "[@metadata][mymessage]" ]
	}

	# parse the json (should contain logType)
	json
	{
		id => "json"
		source => "[@metadata][mymessage]"
	}

	# check if we had parsing errors. If we had any errors, keep the original input for debugging, otherise delete it
	if "_jsonparsefailure" in [tags]
	{
		mutate
		{
			id => "keep_original_message"
			# if logstash was not able to parse the json, kepp original message for debugging purpose. Otherwise get rid of it
			rename => ["[@metadata][mymessage]", "[logstash][debug][originalMessage]" ]
		}
	}
}

```

It works fine if I get a valid json. But if I provoke a \_jsonparsefailure I the field `logstash.debug.originalMessage` is `empty` and I have following error in logstash's log:

`[2019-11-15T08:21:55,162][WARN][logstash.filters.json][generic_json] Parsed JSON object/hash requires a target configuration option {:source=>"[@metadata][mymessage]", :raw=>"\"das ist ein parsing fehler2\" "}`

If I use a normal field instead of @metadata it works like charm. It gets renamed and the content stays and I remove the field if no parsing error is tagged.

But I want to understand why these behaviors are different 😉

Thanks, Andreas

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 15, 2019, 3:13pm UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028/2 "2019-11-15T15:13:07Z")

</div>

You are hitting [this](https://github.com/logstash-plugins/logstash-filter-mutate/issues/77) bug.

---

<div class="post-metadata">

### Author: ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)
#### Post date: [November 18, 2019, 12:26pm UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028/3 "2019-11-18T12:26:57Z")

</div>

Hi @Badger,

Thanks.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2019, 12:26pm UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028/4 "2019-12-16T12:26:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.