I have this JSON log message (from logstash-logback-encoder):
{
"@version" => "1",
"@timestamp" => 2018-05-24T09:49:43.336Z,
"message" => "{
"@timestamp":"2018-05-24T11:49:40.521+02:00",
"thread":"container-1",
"logger_name":"it.parsing.MemoParser",
"appName":"memo-ingestor",
"level":"DEBUG",
"message":" **** Matched Records 40000"}",
"tags" => [
[0] "_grokparsefailure"
],
"port" => 43946,
"host" => "it.myhost.intranet"
}
}
In my logstash.conf I've added this configuration:
mutate {
add_field => {
"logLevel" => '%{[message][level]}'
"appName" => '%{[message][appName]}'
}
}
but new fields are:
"appName": "%{[message][appName]}",
"logLevel": "%{[message][level]}",
How can I extract the appName and level fields from the message field?
Please show your complete configuration. Are you sending events between Logstash instances?
Hi @magnusbaeck ,
here is my configuration in logback.xml:
<appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>${logstash.tcp.ip}:${logstash.tcp.port}</destination>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<timeZone>Europe/Berlin</timeZone>
</timestamp>
<threadName>
<fieldName>thread</fieldName>
</threadName>
<loggerName />
<pattern>
<pattern>
{ "appName": "my app" }
</pattern>
</pattern>
<logLevel />
<message />
<stackTrace />
</providers>
</encoder>
</appender>
I resolved my problem with this configuration in logstash.conf:
input {
tcp {
port => 5000
}
}
filter {
json {
source => "message"
target => "logInfo"
}
mutate {
add_field => {
"level" => "%{[logInfo][level]}"
"appName" => "%{[logInfo][appName]}"
"thread" => "%{[logInfo][thread]}"
"loggerName" => "%{[logInfo][logger_name]}"
"logMessage" => "%{[logInfo][message]}"
"logtimestamp" => "%{[logInfo][@timestamp]}"
}
}
if [logInfo][stack_trace] {
mutate {
add_field => {
"logStackTrace" => "%{[logInfo][stack_trace]}"
}
}
}
mutate {
remove_field => [ "logInfo" ]
remove_field => [ "message" ]
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
elasticsearch {
hosts => "elasticsearch:9200"
index => "logstash-%{+YYYY.MM}"
}
}
I still have some doubts, in particular I would like to show video (to debug) the input log message from logback to logstash.
Why not remove the target
option from your json filter? Then you won't have to move the fields from [logInfo][X]
to X
and so on.
@magnusbaeck
I'm still not able to understand logstash completely, so I proceeded to attempts.
Before these changes, in the output message, I had:
...
_source: {
message: {level: "myloglevel", message: "mylogmessage", (and so on ...)}
}
...
if I remove target, as you say, I am the same able to move the fields directly under the _source field?
...
_source: {
message: "mylogmessage",
level: "myloglevel",
...
}
...
I hope you understand my doubt
You won't have to move them. That's the point. They will be placed at the top level from the start. Try it out.
Ok, I removed target and other instructions to move fields:
The output is OK!
But now I ask you: will nested app log @timestamp replace logstash @timestamp ?
ok, I verified that the timestamp is that of the application log
Many thanks !!!
system
(system)
Closed
June 22, 2018, 8:45am
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.