Json parsin

chrispos · March 2, 2023, 12:51pm

Hello,
I have a question. We are trying to set up a logging system for a java application running on Jboss. The goal is to be able to filter for certain errors. We've done the following

Server.log converted to server.json (Jboss server this was an option in the software)
Forward server.json to logstash with the following filebeat config.

filebeat.inputs:
- type: log
    enabled: true
    paths:
      - /opt/rh/eap7/root/usr/share/wildfly/standalone/log/server.json
output.logstash:
    hosts: ["192.XX.XX.XXX:5044"]

The logstash configuration on the remote server looks like this.

input {
    beats {
        port => "5044"
    }
}
filter {
    json {
        source => "message"
    }
}
output{
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "data"
    }
}
~

This is the JSON produced by Jboss

{
      "timestamp": "2023-03-01T00:00:16.739+01:00",
      "series": 6760,
      "loggerClassName": "org.apache.commons.logging.impl.JBossLog",
      "loggerName": "org.springframework.boot.actuate.ldap.LdapHealthIndicator",
      "level": "WARNING",
      "message": "LDAP health check failed",
      "threadName": "default task-2",
      "threadId": 193,
      "mdc": {
      },
      "ndc": "",
      "hostName": "idontknow.youdo.com",
      "processName": "jboss-modules.jar",
      "processId": 23053,
      "stackTrace": ":org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occurred during blablalbla…..

Now I receive data neatly and visible in Kibana, but I am not able to filter correctly. Like in this youtube video (Finding insights and taking action in Discover - YouTube) I think there is something wrong with my configuration.
We would like filter our server.json ; timestamp, level: WARN or ERROR, message: what the error message is.

I am looking forward for your answer.

leandrojmp · March 2, 2023, 2:40pm

What is the output you are getting In Logstash/Elastic? It is not clear what is your output.

Also, is your json file pretty printed or it is one json event per line?

chrispos · March 2, 2023, 3:05pm

Hello thanks for your message.

Can you explain to me where I can see what the output is? (excuse me I used to be a physiotherapist but I already switched to IT, so I'm still a beginner)

leandrojmp · March 2, 2023, 7:29pm

For example, what is the output you have in Elasticsearch/Kibana?

Rios · March 2, 2023, 10:17pm

Chris can you also show a message in JSON format inside Kibana?
On the left side should be visible list of fields, also screen that.

According to this which you are mentioned, the log is in the JSON format, then transformed and send to ES. If all fields are parsed, you should be able to do filtering.

chrispos · March 7, 2023, 12:22pm

Hello,

Thanks for your reply.

I hope this is what your looking for

{ "_index": "data", "_id": "c4LUMYYBOrQQ_SwCUqAx", "_version": 1, "_score": 0, "_source": { "@timestamp": "2023-02-08T16:20:42.577Z", "ecs": { "version": "8.0.0" }, "event": { "original": " \"level\": \"ERROR\"," }, "@version": "1", "input": { "type": "log" }, "agent": { "type": "filebeat", "version": "8.5.3", "id": "43aa7691-83e1-4a82-8295-1823096cd549", "ephemeral_id": "ea5c1d98-5130-49b8-986f-0483129ec4fb", "name": "fictionserver" }, "message": " \"level\": \"ERROR\",", "tags": [ "beats_input_codec_plain_applied", "_jsonparsefailure" ], "host": { "name": "fictionserver" }, "log": { "file": { "path": "/log/server.json" }, "offset": 17955389 } }, "fields": { "agent.version.keyword": [ "8.5.3" ], "input.type.keyword": [ "log" ], "host.name.keyword": [ "fictionserver" ], "tags.keyword": [ "beats_input_codec_plain_applied", "_jsonparsefailure" ], "agent.type": [ "filebeat" ], "ecs.version.keyword": [ "8.0.0" ], "@version": [ "1" ], "agent.name": [ "fictionserver" ], "host.name": [ "fictionserver" ], "log.file.path.keyword": [ "/log/server.json" ], "agent.type.keyword": [ "filebeat" ], "agent.ephemeral_id.keyword": [ "ea5c1d98-5130-49b8-986f-0483129ec4fb" ], "event.original": [ " \"level\": \"ERROR\"," ], "agent.name.keyword": [ "fictionserver" ], "agent.id.keyword": [ "43aa7691-83e1-4a82-8295-1823096cd549" ], "input.type": [ "log" ], "@version.keyword": [ "1" ], "log.offset": [ 17955389 ], "message": [ " \"level\": \"ERROR\"," ], "tags": [ "beats_input_codec_plain_applied", "_jsonparsefailure" ], "@timestamp": [ "2023-02-08T16:20:42.577Z" ], "agent.id": [ "43aa7691-83e1-4a82-8295-1823096cd549" ], "ecs.version": [ "8.0.0" ], "message.keyword": [ " \"level\": \"ERROR\"," ], "log.file.path": [ "/log/server.json" ], "event.original.keyword": [ " \"level\": \"ERROR\"," ], "agent.ephemeral_id": [ "ea5c1d98-5130-49b8-986f-0483129ec4fb" ], "agent.version": [ "8.5.3" ] } }

Rios · March 8, 2023, 9:14am

[ " \"level\": \"ERROR\"," ], "tags": [ "beats_input_codec_plain_applied", "_jsonparsefailure" ],

You have data which is not a valid JSON format.

It seems, you have removed the message field, search for "_id": "c4LUMYYBOrQQ_SwCUqAx" check or there is a line which contains only : \"level\": \"ERROR\"
Inspect the same file in a editor.

chrispos · March 13, 2023, 4:20pm

I've become a bit wiser now, but the problem seems bigger. I may have run filebeat twice under 2 accounts. (if that is even possible)
It seems that there are problems with json structure as mentioned. Now when I run logstash on the server I get the following error

[2023-03-13T16:55:20,897][WARN ][logstash.filters.json ][main][75aaa507d5ca28b531d4a5cbc1c2758e85b01e119ec6a7b9d0b6dcf92b65e39e] Error parsing json {:source=>"message", :raw=>" \"processName\": \"jboss-modules.jar\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') )

If I run a cat on the server.json on the server and copy it to an online json validator it works fine on several websites. Where is my problem now, in my json or in my filebeat or logstash configuration.

This is the cat from the server.json


{
    "timestamp": "2023-03-13T15:44:41.86+01:00",
    "sequence": 155,
    "loggerClassName": "org.jboss.as.repository.logging.DeploymentRepositoryLogger_$logger",
    "loggerName": "org.jboss.as.repository",
    "level": "INFO",
    "message": "WFLYDR0002: Content removed from location /opt/content/7a/913bd056d5ab73c08152c4d1066e1353002298/content",
    "threadName": "ServerService Thread Pool -- 38",
    "threadId": 61,
    "mdc": {
    },
    "ndc": "",
    "hostName": "idontknwXXX",
    "processName": "jboss-modules.jar",
    "processId": 38267
}

Badger · March 13, 2023, 5:59pm

logstash is processing a single line from the file. You need to configure filebeat to combine every line of the pretty-printed JSON object into a single event using the multiline feature.

chrispos · March 20, 2023, 3:46pm

Thanks for your message. I do understand the principle of the multilines. But I have no idea how to apply it to my own server.json and where to start. I almost don't dare to ask, but can someone provide an example image for my server.json regarding the multilines principle.

Thank you in advance

Badger · March 20, 2023, 4:34pm

I suggest you ask in the filebeat forum, its multiline feature is rather different to that in logstash.

system · April 17, 2023, 4:35pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.