Log4j 2 json parsing with logstash 1.4.2

Elastic Stack Logstash
1

I am using log4j 2 along with json pattern layout to create the logs and using the logstash agent 1.4.2 to port the logs in to a redis. In my logs i am trying to log some json payloads which includes some nested json. Also i am using json filter to parse the json. My issue is when parsing the json into fields is not happening properly.

My issues are

  1. flat json is not parsed unless two json filters added
  2. nested jsons will not be parsed and even 2 json filters are appears

below are my configurations

  1. logstash shipper.conf

input {
file {
tags => ["messaging-stg"]
path => "/opt/messaging/logs/logs.log"
debug => true
type => "messaging-stg"
}
}
filter {
if [type]== "messaging-stg" {
json {
source => "message"
}
# first filter parse entire json message to a "field" called "message" from "_source"
json {
source => "message"
}
# second filter parse flat json message to fields
}
}
output {
redis {
host => [
"server_01" ,
"server_02"
]
shuffle_hosts => true
data_type => "list"
key => "logstash"
}
}

  1. log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>

<Configuration status="info">

<properties>

<property name="pattern">%m%n</property>

<property name="filePath">/opt/messaging/logs</property>

<property name="fileName">logs</property>

</properties>

<Appenders>

<RollingFile name="RollingFile" fileName="${filePath}/${fileName}.log"

filePattern="${filePath}/${fileName}-%d{yyyy-MM-dd}-%i.log" append="true">

<JSONLayout complete="false" compact="true" eventEol="true" charset="UTF-8" />

<PatternLayout>

<pattern>${pattern}</pattern>

</PatternLayout>

<Policies>

<SizeBasedTriggeringPolicy size="10 KB"/>

</Policies>l

</RollingFile>

<Console name="STDOUT" target="SYSTEM_OUT">

<PatternLayout>

<pattern>${pattern}</pattern>

</PatternLayout>

</Console>

</Appenders>

<Loggers>

<Root level="debug">

<AppenderRef ref="RollingFile"/>

<AppenderRef ref="STDOUT"/>

</Root>

</Loggers>
</Configuration>

check for following sample json

  1. success full flat json parsing
    {
    "Msgtype": "SentToxxxxxxxx",
    "ActionType": "xxxxxx",
    "userstd": {
    "user": {
    "userproperty": {
    "lastupdate": "",
    "createdate": "",
    "propertyvalue": "",
    "propertyname": ""
    },
    "clientid": 444444,
    "lastupdate": "2016-08-25T03:30:09.067",
    "emailaddress": "xxxxxx@xxxxxx.com",
    "userid": 4444444,
    "createdate": "2016-08-25T03:30:09.067",
    "action": "xxxxxxx",
    "lastname": "xxxxxxx",
    "loginid": "xxxxx",
    "firstname": "xxxxx",
    "password": "xxxxxxxx"
    },
    "version": ""
    }
    }

  2. unsuccessfull nested json parsing
    {
    "Msgtype": "Receivedfromxxxxx",
    "ActionType": "xxxxxx",
    "userstd": {
    "user": {
    "clientid": 4444444,
    "lastupdate": "",
    "emailaddress": "",
    "createdate": "",
    "userid": 4444444,
    "lastname": "",
    "enrollment": {
    "lastupdate": "",
    "enrolltype": "course",
    "roleid": 1,
    "createdate": "2016-08-24T22:37:55.555",
    "clientnodeid": 4444444,
    "comment": ""
    },
    "firstname": "",
    "loginid": "",
    "password": ""
    },
    "version": ""
    }
    }

2

Please edit your post and move it to the Logstash category. It doesn't belong in
the Elastic User Groups category.

1 Like
3

Hi , Thank you very much for your information.Sorry for making trouble.

4

HI, Anyone can help over my issue it would appreciated.

5

HI Magnus , iffel that i was able to narrow down my issue,but still having the issue.once i run the logstash by pointing the logs file which needed to parsed i was able to get the proper break down of the json message into fields.It means that the way i am configured the input as file is not in properly configured.

steps carried out to trouble shoot the issues.

  1. downloaded my log4j2 log files and run the logstash with a test config file as below
    ./bin/logstash -f test.config < logs.log this gave me proper out put to the console.But i am wondering why the input type file does not happens properly.can u help me over this.

-------------------------------test.config file ------------------------------------
input { stdin { } }

filter {
json {
source => "message"
}
json {
source => "message"
}
}
output { stdout { codec => rubydebug } }