# Extract subfields in JSON Field

**URL:** <https://discuss.elastic.co/t/extract-subfields-in-json-field/133111>\
**Category:** Logstash\
**Created:** [May 24, 2018, 10:00am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111 "2018-05-24T10:00:54Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alessio\_Frabotta](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\
**Post date:** [May 24, 2018, 10:00am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/1 "2018-05-24T10:00:54Z")

</div>

I have this JSON log message (from logstash-logback-encoder):

```
{
    "@version" => "1",
    "@timestamp" => 2018-05-24T09:49:43.336Z,
    "message" => "{
        "@timestamp":"2018-05-24T11:49:40.521+02:00",
        "thread":"container-1",
        "logger_name":"it.parsing.MemoParser",
        "appName":"memo-ingestor",
        "level":"DEBUG",
        "message":" **** Matched Records 40000"}",
        "tags" => [
            [0] "_grokparsefailure"
         ],
         "port" => 43946,
         "host" => "it.myhost.intranet"
    }
}  	

```

In my logstash.conf I've added this configuration:

```
mutate {
		add_field => {
			"logLevel" => '%{[message][level]}'
			"appName" => '%{[message][appName]}'
		}
	}

```

but new fields are:

```
"appName": "%{[message][appName]}",
"logLevel": "%{[message][level]}",

```

How can I extract the appName and level fields from the message field?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 5:54am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/2 "2018-05-25T05:54:14Z")

</div>

Please show your complete configuration. Are you sending events between Logstash instances?

---

<div class="post-metadata">

**Author:** ![Alessio\_Frabotta](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\
**Post date:** [May 25, 2018, 7:43am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/3 "2018-05-25T07:43:18Z")

</div>

Hi @magnusbaeck,

here is my configuration in logback.xml:

```
 <appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>${logstash.tcp.ip}:${logstash.tcp.port}</destination>

		<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
			<providers>
				<timestamp>
					<timeZone>Europe/Berlin</timeZone>
				</timestamp>
				<threadName>
					<fieldName>thread</fieldName>
				</threadName>
				<loggerName />
			        <pattern>
	                <pattern> 
	                        { "appName": "my app" }
	                    </pattern>
                   </pattern>
			       <logLevel />
			       <message />
			       <stackTrace />
		      </providers>
	      </encoder>
</appender>

```

**I resolved my problem with this configuration in logstash.conf:**

```
input {
	tcp {
    	port => 5000
    }
}

filter {

	json {
    	source => "message"
  		target => "logInfo" 		
  	}

	mutate {
		add_field => {
			"level" => "%{[logInfo][level]}"
			"appName" => "%{[logInfo][appName]}"
			"thread" => "%{[logInfo][thread]}"
			"loggerName" => "%{[logInfo][logger_name]}"
			"logMessage" => "%{[logInfo][message]}"
			"logtimestamp" => "%{[logInfo][@timestamp]}"
		}
	}
	
	if [logInfo][stack_trace] {
		mutate {
			add_field => {
				"logStackTrace" => "%{[logInfo][stack_trace]}"
			}
		}
	}
	
	mutate {
    	remove_field => ["logInfo"]
    	remove_field => ["message"]
  	}
}

output {
	stdout {
  		codec => rubydebug { metadata => true }
  	}
	
  	elasticsearch {
    	hosts => "elasticsearch:9200"
    	index => "logstash-%{+YYYY.MM}"
  	}
}

```

I still have some doubts, in particular I would like to show video (to debug) the input log message from logback to logstash.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 7:45am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/4 "2018-05-25T07:45:23Z")

</div>

Why not remove the `target` option from your json filter? Then you won't have to move the fields from `[logInfo][X]` to `X` and so on.

---

<div class="post-metadata">

**Author:** ![Alessio\_Frabotta](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\
**Post date:** [May 25, 2018, 8:08am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/5 "2018-05-25T08:08:32Z")

</div>

@magnusbaeck  
I'm still not able to understand logstash completely, so I proceeded to attempts.

Before these changes, in the output message, I had:

```
...
_source: {
    message: {level: "myloglevel", message: "mylogmessage", (and so on ...)}
}
...

```

if I remove target, as you say, I am the same able to move the fields directly under the \_source field?

```
...
_source: {
    message: "mylogmessage",
    level: "myloglevel",
    ...
}
...

```

I hope you understand my doubt 🤔

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 8:17am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/6 "2018-05-25T08:17:23Z")

</div>

You won't have to move them. That's the point. They will be placed at the top level from the start. Try it out.

---

<div class="post-metadata">

**Author:** ![Alessio\_Frabotta](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\
**Post date:** [May 25, 2018, 8:27am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/7 "2018-05-25T08:27:06Z")

</div>

Ok, I removed target and other instructions to move fields:  
The output is OK!  
But now I ask you: will nested app log @timestamp replace logstash @timestamp?

---

<div class="post-metadata">

**Author:** ![Alessio\_Frabotta](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\
**Post date:** [May 25, 2018, 8:45am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/8 "2018-05-25T08:45:46Z")

</div>

ok, I verified that the timestamp is that of the application log

Many thanks !!! 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2018, 8:45am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111/9 "2018-06-22T08:45:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.