Process logs in json with a "message" key not working with filebeat 6.4.1

Elastic Stack Beats
1

Hello,

I have an app that produce logs in that form :
{
"time": "2018-10-01T16:59:33+02:00",
"level": "INFO",
"component_name": "test",
"request_type": "incomming_request",
"request_peer": "front",
"session_id": "todo",
"request_id": "todo",
"request_method": "POST",
"request_uri": "/setconfig",
"request_status_code": "200",
"message": {
"remote_ip": "1.1.1.1",
"timeout": "10"
}
}

My filebeat config :
- paths:
- "output.log"
json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true
type: log

I am aware that "message" key is used in the Elastic Common Schema thats why i use the "json.overwrite_keys: true" option.

But i still have theses warnings in filebeat logs {"type":"mapper_parsing_exception","reason":"failed to parse [message]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:396"} and the events are not sent to elasticsearch.

Is that a normal behaviour ? With the overwrite key set to true this shouldnt be a problem ?

2

If you modify the type of an existing field, you need to update your index mapping in Elasticsearch, because it still expects message field to be a string. Thus, indexing fails.

3

I dont use custom maping, i was thinking filebeat was sending one to ES with the right types acording to his configuration (in my case json object)

So for my case is better to go with a custom template and specify the type of message ?

4

Exactly. Unfortunately, Filebeat comes with a default generated mapping based on the official exported fields. It does not take into account the possible modifications by users.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.