Extract subfields in JSON Field

[Alessio_Frabotta]

I have this JSON log message (from logstash-logback-encoder):

{
    "@version" => "1",
    "@timestamp" => 2018-05-24T09:49:43.336Z,
    "message" => "{
        "@timestamp":"2018-05-24T11:49:40.521+02:00",
        "thread":"container-1",
        "logger_name":"it.parsing.MemoParser",
        "appName":"memo-ingestor",
        "level":"DEBUG",
        "message":" **** Matched Records 40000"}",
        "tags" => [
            [0] "_grokparsefailure"
         ],
         "port" => 43946,
         "host" => "it.myhost.intranet"
    }
}  	

In my logstash.conf I've added this configuration:

mutate {
		add_field => {
			"logLevel" 	=> '%{[message][level]}'
			"appName" 	=> '%{[message][appName]}'
		}
	}

but new fields are:

"appName": "%{[message][appName]}",
"logLevel": "%{[message][level]}",

How can I extract the appName and level fields from the message field?

[magnusbaeck (Magnus Bäck)]

Please show your complete configuration. Are you sending events between Logstash instances?

[Alessio_Frabotta]

Hi @magnusbaeck,

here is my configuration in logback.xml:

 <appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>${logstash.tcp.ip}:${logstash.tcp.port}</destination>

		<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
			<providers>
				<timestamp>
					<timeZone>Europe/Berlin</timeZone>
				</timestamp>
				<threadName>
					<fieldName>thread</fieldName>
				</threadName>
				<loggerName />
			        <pattern>
	                <pattern> 
	                        { "appName": "my app" }
	                    </pattern>
                   </pattern>
			       <logLevel />
			       <message />
			       <stackTrace />
		      </providers>
	      </encoder>
</appender>

I resolved my problem with this configuration in logstash.conf:

input {
	tcp {
    	port => 5000
    }
}

filter {

	json {
    	source => "message"
  		target => "logInfo" 		
  	}

	mutate {
		add_field => {
			"level" 			=> "%{[logInfo][level]}"
			"appName" 			=> "%{[logInfo][appName]}"
			"thread" 			=> "%{[logInfo][thread]}"
			"loggerName" 		=> "%{[logInfo][logger_name]}"
			"logMessage" 		=> "%{[logInfo][message]}"
			"logtimestamp" 		=> "%{[logInfo][@timestamp]}"
		}
	}
	
	if [logInfo][stack_trace] {
		mutate {
			add_field => {
				"logStackTrace" => "%{[logInfo][stack_trace]}"
			}
		}
	}
	
	mutate {
    	remove_field => [ "logInfo" ]
    	remove_field => [ "message" ]
  	}
}

output {
	stdout {
  		codec => rubydebug { metadata => true }
  	}
	
  	elasticsearch {
    	hosts => "elasticsearch:9200"
    	index => "logstash-%{+YYYY.MM}"
  	}
}

I still have some doubts, in particular I would like to show video (to debug) the input log message from logback to logstash.

[magnusbaeck (Magnus Bäck)]

Why not remove the target option from your json filter? Then you won't have to move the fields from [logInfo][X] to X and so on.

[Alessio_Frabotta]

@magnusbaeck
I'm still not able to understand logstash completely, so I proceeded to attempts.

Before these changes, in the output message, I had:

...
_source: {
    message: {level: "myloglevel", message: "mylogmessage", (and so on ...)}
}
...

if I remove target, as you say, I am the same able to move the fields directly under the _source field?

...
_source: {
    message: "mylogmessage",
    level: "myloglevel",
    ...
}
...

I hope you understand my doubt

[magnusbaeck (Magnus Bäck)]

You won't have to move them. That's the point. They will be placed at the top level from the start. Try it out.

[Alessio_Frabotta]

Ok, I removed target and other instructions to move fields:
The output is OK!
But now I ask you: will nested app log @timestamp replace logstash @timestamp?

[Alessio_Frabotta]

ok, I verified that the timestamp is that of the application log

Many thanks !!!