# Json parsin

**URL:** <https://discuss.elastic.co/t/json-parsin/326849>\
**Category:** Logstash\
**Created:** [March 2, 2023, 12:51pm UTC](https://discuss.elastic.co/t/json-parsin/326849 "2023-03-02T12:51:34Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [March 2, 2023, 12:51pm UTC](https://discuss.elastic.co/t/json-parsin/326849/1 "2023-03-02T12:51:34Z")

</div>

Hello,  
I have a question. We are trying to set up a logging system for a java application running on Jboss. The goal is to be able to filter for certain errors. We've done the following

- Server.log converted to server.json (Jboss server this was an option in the software)
- Forward server.json to logstash with the following filebeat config.

```auto
filebeat.inputs:
- type: log
    enabled: true
    paths:
      - /opt/rh/eap7/root/usr/share/wildfly/standalone/log/server.json
output.logstash:
    hosts: ["192.XX.XX.XXX:5044"]
```

The logstash configuration on the remote server looks like this.

```auto
input {
    beats {
        port => "5044"
    }
}
filter {
    json {
        source => "message"
    }
}
output{
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "data"
    }
}
~
```

This is the JSON produced by Jboss

```auto
{
      "timestamp": "2023-03-01T00:00:16.739+01:00",
      "series": 6760,
      "loggerClassName": "org.apache.commons.logging.impl.JBossLog",
      "loggerName": "org.springframework.boot.actuate.ldap.LdapHealthIndicator",
      "level": "WARNING",
      "message": "LDAP health check failed",
      "threadName": "default task-2",
      "threadId": 193,
      "mdc": {
      },
      "ndc": "",
      "hostName": "idontknow.youdo.com",
      "processName": "jboss-modules.jar",
      "processId": 23053,
      "stackTrace": ":org.springframework.ldap.UncategorizedLdapException: Uncategorized exception occurred during blablalbla…..
```

Now I receive data neatly and visible in Kibana, but I am not able to filter correctly. Like in this youtube video ([Finding insights and taking action in Discover - YouTube](https://youtu.be/rsMlbNGUlIM?t=136)) I think there is something wrong with my configuration.  
We would like filter our server.json ; timestamp, level: WARN or ERROR, message: what the error message is.

I am looking forward for your answer.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 2, 2023, 2:40pm UTC](https://discuss.elastic.co/t/json-parsin/326849/2 "2023-03-02T14:40:26Z")

</div>

What is the output you are getting In Logstash/Elastic? It is not clear what is your output.

Also, is your json file pretty printed or it is one json event per line?

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [March 2, 2023, 3:05pm UTC](https://discuss.elastic.co/t/json-parsin/326849/3 "2023-03-02T15:05:19Z")

</div>

Hello thanks for your message.

Can you explain to me where I can see what the output is? (excuse me I used to be a physiotherapist but I already switched to IT, so I'm still a beginner)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 2, 2023, 7:29pm UTC](https://discuss.elastic.co/t/json-parsin/326849/4 "2023-03-02T19:29:25Z")

</div>

For example, what is the output you have in Elasticsearch/Kibana?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [March 2, 2023, 10:17pm UTC](https://discuss.elastic.co/t/json-parsin/326849/5 "2023-03-02T22:17:01Z")

</div>

Chris can you also show a message in JSON format inside Kibana?  
On the left side should be visible list of fields, also screen that.

According to this which you are mentioned, the log is in the JSON format, then transformed and send to ES. If all fields are parsed, you should be able to do filtering.

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [March 7, 2023, 12:22pm UTC](https://discuss.elastic.co/t/json-parsin/326849/6 "2023-03-07T12:22:47Z")

</div>

Hello,

Thanks for your reply.

I hope this is what your looking for

`
{
  "_index": "data",
  "_id": "c4LUMYYBOrQQ_SwCUqAx",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-02-08T16:20:42.577Z",
    "ecs": {
      "version": "8.0.0"
    },
    "event": {
      "original": " \"level\": \"ERROR\","
    },
    "@version": "1",
    "input": {
      "type": "log"
    },
    "agent": {
      "type": "filebeat",
      "version": "8.5.3",
      "id": "43aa7691-83e1-4a82-8295-1823096cd549",
      "ephemeral_id": "ea5c1d98-5130-49b8-986f-0483129ec4fb",
      "name": "fictionserver"
    },
    "message": " \"level\": \"ERROR\",",
    "tags": [
      "beats_input_codec_plain_applied",
      "_jsonparsefailure"
    ],
    "host": {
      "name": "fictionserver"
    },
    "log": {
      "file": {
        "path": "/log/server.json"
      },
      "offset": 17955389
    }
  },
  "fields": {
    "agent.version.keyword": [
      "8.5.3"
    ],
    "input.type.keyword": [
      "log"
    ],
    "host.name.keyword": [
      "fictionserver"
    ],
    "tags.keyword": [
      "beats_input_codec_plain_applied",
      "_jsonparsefailure"
    ],
    "agent.type": [
      "filebeat"
    ],
    "ecs.version.keyword": [
      "8.0.0"
    ],
    "@version": [
      "1"
    ],
    "agent.name": [
      "fictionserver"
    ],
    "host.name": [
      "fictionserver"
    ],
    "log.file.path.keyword": [
      "/log/server.json"
    ],
    "agent.type.keyword": [
      "filebeat"
    ],
    "agent.ephemeral_id.keyword": [
      "ea5c1d98-5130-49b8-986f-0483129ec4fb"
    ],
    "event.original": [
      " \"level\": \"ERROR\","
    ],
    "agent.name.keyword": [
      "fictionserver"
    ],
    "agent.id.keyword": [
      "43aa7691-83e1-4a82-8295-1823096cd549"
    ],
    "input.type": [
      "log"
    ],
    "@version.keyword": [
      "1"
    ],
    "log.offset": [
      17955389
    ],
    "message": [
      " \"level\": \"ERROR\","
    ],
    "tags": [
      "beats_input_codec_plain_applied",
      "_jsonparsefailure"
    ],
    "@timestamp": [
      "2023-02-08T16:20:42.577Z"
    ],
    "agent.id": [
      "43aa7691-83e1-4a82-8295-1823096cd549"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "message.keyword": [
      " \"level\": \"ERROR\","
    ],
    "log.file.path": [
      "/log/server.json"
    ],
    "event.original.keyword": [
      " \"level\": \"ERROR\","
    ],
    "agent.ephemeral_id": [
      "ea5c1d98-5130-49b8-986f-0483129ec4fb"
    ],
    "agent.version": [
      "8.5.3"
    ]
  }
}`

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [March 8, 2023, 9:14am UTC](https://discuss.elastic.co/t/json-parsin/326849/7 "2023-03-08T09:14:04Z")

</div>

`[" \"level\": \"ERROR\","], "tags": ["beats_input_codec_plain_applied", "_jsonparsefailure"],`

You have data which is not a valid JSON format.

It seems, you have removed the message field, search for "\_id": "c4LUMYYBOrQQ\_SwCUqAx" check or there is a line which contains only : `\"level\": \"ERROR\"`  
Inspect the same file in a editor.

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [March 13, 2023, 4:20pm UTC](https://discuss.elastic.co/t/json-parsin/326849/8 "2023-03-13T16:20:39Z")

</div>

I've become a bit wiser now, but the problem seems bigger. I may have run filebeat twice under 2 accounts. (if that is even possible)  
It seems that there are problems with json structure as mentioned. Now when I run logstash on the server I get the following error

`
[2023-03-13T16:55:20,897][WARN][logstash.filters.json][main][75aaa507d5ca28b531d4a5cbc1c2758e85b01e119ec6a7b9d0b6dcf92b65e39e] Error parsing json {:source=>"message", :raw=>" \"processName\": \"jboss-modules.jar\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
)`

If I run a cat on the server.json on the server and copy it to an online json validator it works fine on several websites. Where is my problem now, in my json or in my filebeat or logstash configuration.

This is the cat from the server.json

```auto

{
    "timestamp": "2023-03-13T15:44:41.86+01:00",
    "sequence": 155,
    "loggerClassName": "org.jboss.as.repository.logging.DeploymentRepositoryLogger_$logger",
    "loggerName": "org.jboss.as.repository",
    "level": "INFO",
    "message": "WFLYDR0002: Content removed from location /opt/content/7a/913bd056d5ab73c08152c4d1066e1353002298/content",
    "threadName": "ServerService Thread Pool -- 38",
    "threadId": 61,
    "mdc": {
    },
    "ndc": "",
    "hostName": "idontknwXXX",
    "processName": "jboss-modules.jar",
    "processId": 38267
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 13, 2023, 5:59pm UTC](https://discuss.elastic.co/t/json-parsin/326849/9 "2023-03-13T17:59:49Z")

</div>

> [@chrispos](#):
>
> :raw=\>" "processName": "jboss-modules.jar",",

logstash is processing a single line from the file. You need to configure filebeat to combine every line of the pretty-printed JSON object into a single event using the [multiline](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) feature.

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [March 20, 2023, 3:46pm UTC](https://discuss.elastic.co/t/json-parsin/326849/10 "2023-03-20T15:46:12Z")

</div>

Thanks for your message. I do understand the principle of the multilines. But I have no idea how to apply it to my own server.json and where to start. I almost don't dare to ask, but can someone provide an example image for my server.json regarding the multilines principle.

Thank you in advance

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 20, 2023, 4:34pm UTC](https://discuss.elastic.co/t/json-parsin/326849/11 "2023-03-20T16:34:54Z")

</div>

I suggest you ask in the filebeat forum, its multiline feature is rather different to that in logstash.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2023, 4:35pm UTC](https://discuss.elastic.co/t/json-parsin/326849/12 "2023-04-17T16:35:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.