# Process logs in json with a "message" key not working with filebeat 6.4.1

**URL:** https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688
**Category:** Beats
**Tags:** filebeat
**Created:** [October 2, 2018, 11:42am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688 "2018-10-02T11:42:52Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![MatMatMatMatMatMat](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@MatMatMatMatMatMat](https://discuss.elastic.co/u/MatMatMatMatMatMat)
#### Post date: [October 2, 2018, 11:42am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688/1 "2018-10-02T11:42:52Z")

</div>

Hello,

I have an app that produce logs in that form :  
{  
"time": "2018-10-01T16:59:33+02:00",  
"level": "INFO",  
"component\_name": "test",  
"request\_type": "incomming\_request",  
"request\_peer": "front",  
"session\_id": "todo",  
"request\_id": "todo",  
"request\_method": "POST",  
"request\_uri": "/setconfig",  
"request\_status\_code": "200",  
"message": {  
"remote\_ip": "1.1.1.1",  
"timeout": "10"  
}  
}

My filebeat config :  
- paths:  
- "output.log"  
json.keys\_under\_root: true  
json.add\_error\_key: true  
json.overwrite\_keys: true  
type: log

I am aware that "message" key is used in the Elastic Common Schema thats why i use the "json.overwrite\_keys: true" option.

But i still have theses warnings in filebeat logs {"type":"mapper\_parsing\_exception","reason":"failed to parse [message]","caused\_by":{"type":"illegal\_state\_exception","reason":"Can't get text on a START\_OBJECT at 1:396"} and the events are not sent to elasticsearch.

Is that a normal behaviour ? With the overwrite key set to true this shouldnt be a problem ?

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [October 3, 2018, 7:53am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688/2 "2018-10-03T07:53:05Z")

</div>

If you modify the type of an existing field, you need to update your index mapping in Elasticsearch, because it still expects `message` field to be a string. Thus, indexing fails.

---

<div class="post-metadata">

### Author: ![MatMatMatMatMatMat](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@MatMatMatMatMatMat](https://discuss.elastic.co/u/MatMatMatMatMatMat)
#### Post date: [October 3, 2018, 8:17am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688/3 "2018-10-03T08:17:51Z")

</div>

I dont use custom maping, i was thinking filebeat was sending one to ES with the right types acording to his configuration (in my case json object)

So for my case is better to go with a custom template and specify the type of message ?

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [October 4, 2018, 9:38am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688/4 "2018-10-04T09:38:56Z")

</div>

Exactly. Unfortunately, Filebeat comes with a default generated mapping based on the official exported fields. It does not take into account the possible modifications by users.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 1, 2018, 9:39am UTC](https://discuss.elastic.co/t/process-logs-in-json-with-a-message-key-not-working-with-filebeat-6-4-1/150688/5 "2018-11-01T09:39:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
