I'm using Log4J v2.x SocketAppender to send logs to logstash directly. The system was previously designed to send the logs via filebeat. Since I changed the setting, logstash now cannot read my inputs.
[2019-09-26T10:06:16,255][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 2]>, :data=>",\"source\":\"/home/directfn/app/NTP/CloudSetup/jboss-eap-7.1/standalone/log/OMS_Analyzed.log\",\"type\":\"log\"}\r"}
[2019-09-26T10:06:16,256][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: (String)"{"input_type":"log","message":163TXN-DFLT107CACHE_LOAD_FROM_DBOMS-TRADE127.0.0.1trueLOG_MESSAGE:CACHE LOAD FROM DBTXN_TIME:1569472482178DAO:com.dfn.oms.newgen.core.persistance.dao.SysParamDAOCACHE_KEY:SYMBOL_CLEAR_PERIOD"; line: 1, column: 32]>, :data=>"{\"input_type\":\"log\",\"message\":\u001F163\u001FTXN-DFLT\u001F107\u001FCACHE_LOAD_FROM_DB\u001FOMS-TRADE\u001F127.0.0.1\u001Ftrue\u001FLOG_MESSAGE:CACHE LOAD FROM DB\u0001TXN_TIME:1569472482178\u0001DAO:com.dfn.oms.newgen.core.persistance.dao.SysParamDAO\u0001CACHE_KEY:SYMBOL_CLEAR_PERIOD\u0001\u001F"}
I'm sending the logs in UTF-8 format.
Here is the log4j configuration.
<Socket name="SocketRef" host="192.168.0.51" port="5055">
<PatternLayout>
<charset>"UTF-8"</charset>
<pattern>{"input_type":"log","message":"%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %c (%t) %msg","source":"/home/directfn/app/NTP/CloudSetup/jboss-eap-7.1/standalone/log/OMS_Analyzed.log","type":"log"}%n</pattern>
</PatternLayout>
</Socket>
The problem is with the msg since it contains charachters ▼O like this.
Can anyone give me a solution?