# How to use the JSON filter correctly?
**URL:** https://discuss.elastic.co/t/how-to-use-the-json-filter-correctly/339372
**Category:** Logstash
**Created:** [July 27, 2023, 5:02am UTC](https://discuss.elastic.co/t/how-to-use-the-json-filter-correctly/339372 "2023-07-27T05:02:09Z")
**Posts on this page:** 15
**Page:** 1
Application logs is of below JSON format and I'm unsure what should be the **source** field incase I'm using the JSON filter ?
I would like to have all the fields appear on the Kibana output, particularly the message field, but as per the official [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-source) - " For example, if you have JSON data in the `message` field: The above would parse the JSON from the `message` field. "
but I don't have a JSON in my message or any other field but the source is a required setting as well.
How should I handle this ? Thanks in advance.
```auto
{"asctime": "2023-07-26 09:05:08,271", "levelname": "INFO", "pathname": "/app/realtime_processor/parallel_processor.py", "lineno": 280, "message": "comment_body: **SonarQube Quality Gate** \n\n[![Quality Gate passed](https://... does not match defined phrase: rerun ingress-garuda-prod", "namespace": "challenger-rover", "version": "jfrog.artifactory:5060/drpbuilds/list-groovy-real:583eb6a", "pod": "ingress-garuda-prod-647c5d755-7jmwx"}
```
Logstash config -
```auto
input {
http {
port => 9200
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
mutate {
remove_field => ["_index", "_score"]
}
}
output {
elasticsearch {
id => "elk-log-setup"
hosts => [":9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
```
---
> [@Chel\_Db](#):
>
> Application logs is of below JSON format and I'm unsure what should be the **source** field incase I'm using the JSON filter ?
What is your input? You didn't say what is the input you are using.
> [@Chel\_Db](#):
>
> I don't have a JSON in my message or any other field but the source is a required setting as well.
You probably do. When an event enters the logstash pipeline it will be stored on a field named `message`, so you will probably have this:
```auto
{
"message": "{\"asctime\": \"2023-07-26 09:05:08,271\", \"levelname\": \"INFO\", \"pathname\": \"/app/realtime_processor/parallel_processor.py\", \"lineno\": 280, \"message\": \"comment_body: **SonarQube Quality Gate** \\n\\n[![Quality Gate passed](https://... does not match defined phrase: rerun ingress-garuda-prod\", \"namespace\": \"challenger-rover\", \"version\": \"jfrog.artifactory:5060/drpbuilds/list-groovy-real:583eb6a\", \"pod\": \"ingress-garuda-prod-647c5d755-7jmwx\"}"
}
```
In this case the content of the `message` field is your original json log, to parse it you need to use the following filter:
```auto
filter {
json {
source => "message"
}
}
```
Since the event in logstash already has a field named `message`, with your original log, the json filter will override this field with the value of the `message` field in your log.
---
@leandrojmp : My bad, I've added the logstash config to the question.
> "the json filter will override this field with the value of the `message` field in your log."
- Can you please elaborate a bit, you mean the `message` field from my original log, will be retained if I use this config ?
```auto
filter {
json {
source => "message"
}
}
```
Also, all the original fields will be seen in the output (kibana) ?
Thanks !
---
> [@Chel\_Db](#):
>
> Can you please elaborate a bit, you mean the `message` field from my original log, will be retained if I use this config ?
Yes, this is what I said, the `message` field inside your log will override the `message` field **created** by logstash when the event entered the pipeline when the json filter is executed.
Have you tried to see what are the results you will get? The best way to see what you will get is just testing the pipeline.
---
It is not clear what is the issue, the entire log is missing or just the message field from those logs?
If it is just the message field, it is probably because of this:
```auto
json {
source => "message"
remove_field => ["message"]
}
```
Yu have a `remove_field` for the `message` field, normally this will remove the source message, but since your source message also has a message field, it will remove this field.
Remove this `remove_field` from your `json` filter.
---
@leandrojmp : I have attached couple of screenshots of one of the document/event from the dashboard of one of the service.
I hope that provides more context rather than my explanation 🙂
---
Well, this is really confusing.
What is the source of your message? This looks way different from the other examples you shared.
The message in your screenshot looks like a request that you would make to elasticsearch, it is not clear why you are sending this to logstash, can you provide more context?
But if the entire log is missing, did you check logstash logs to see if there is any error?
---
> What is the source of your message? This looks way different from the other examples you shared.
I'm exactly sure what that means, I use fluentd to aggregate logs and funnel that to logstash.
On logstash config, http input plugin is used.
The example logs shared are of same format to that of my production logs.
Below warnings are seen in the logstash logs
```auto
[2023-07-20T00:06:30,436][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"INFO |2023-07-20 00:04:29 |1 | manageQueue.py 335 | Scan details: {'main': {}}", :exception=>#}
[2023-07-20T00:06:35,900][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"INFO |2023-07-20 00:04:46 |1 | manageQueue.py 335 | Scan details: {'main': {}, 'staging': {}}", :exception=>#}
[2023-07-20T00:06:41,900][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"INFO |2023-07-20 00:05:09 |1 | manageQueue.py 324 | There is nothing to process on the queue as it is empty. Returning...", :exception=>#}
[2023-07-20T00:06:46,908][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:38 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:06:52,384][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:42 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:06:57,717][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:45 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:14,423][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:51 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:19,513][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:52 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:25,422][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:52 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:30,430][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:06:53 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:41,453][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:07:12 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:07:57,714][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:07:33 2023 sent heartbeat", :exception=>#}
[2023-07-20T00:08:03,714][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:07:36 2023 sent heartbeat", :exception=>#}
```
---
What I mean is that this is the message you shared in your first post.
```auto
{"asctime": "2023-07-26 09:05:08,271", "levelname": "INFO", "pathname": "/app/realtime_processor/parallel_processor.py", "lineno": 280, "message": "comment_body: **SonarQube Quality Gate** \n\n[![Quality Gate passed](https://... does not match defined phrase: rerun ingress-garuda-prod", "namespace": "challenger-rover", "version": "jfrog.artifactory:5060/drpbuilds/list-groovy-real:583eb6a", "pod": "ingress-garuda-prod-647c5d755-7jmwx"}
```
I assumed that this is what was being sent to Logstash.
But the message in your screenshot is completely different, you have a lot of more fields, and you have what would be multiple messages inside this original `message` field.
So it is not clear what is exactly being sent to Logstash.
It is messages like the first one you shared, or something like the one you shared in your screenshot where you have multiple lines starting with `{"index"`, also you didn't say how are you sending it.
This makes total difference on the troubleshoot, for example if you are using a `Contet-Type` as `application/json` while sending to logstash, this would already parse your message.
Can you provide a sample in plain text of what you are exactly sending to logstash?
---
> [@Chel\_Db](#):
>
> ```auto
> [2023-07-20T00:07:57,714][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:07:33 2023 sent heartbeat", :exception=># at [Source: (byte[])"Thu Jul 20 00:07:33 2023 sent heartbeat"; line: 1, column: 5]>}
> [2023-07-20T00:08:03,714][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"Thu Jul 20 00:07:36 2023 sent heartbeat", :exception=># at [Source: (byte[])"Thu Jul 20 00:07:36 2023 sent heartbeat"; line: 1, column: 5]>}
>
> ```
These are expected, the field `message` does not have a json, so the json filter will obviously not work.
---