# Filebeat shipping incomplete logs
**URL:** https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296
**Category:** Beats
**Tags:** filebeat
**Created:** [June 25, 2019, 9:50am UTC](https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296 "2019-06-25T09:50:00Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@svanschooten](https://discuss.elastic.co/u/svanschooten)
#### Post date: [June 25, 2019, 9:50am UTC](https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296/1 "2019-06-25T09:50:00Z")
We want to improve our logging (filebeat -\> logstash -\> elasticsearch -\> kibana) and so far I have filebeat running on our docker host in its own container (I map all log files manually, since we do not manage the docker host we do not know for sure which containers will be running).
The pipeline logstash -\> elasticsearch -\> kibana is a blackbox, so all I can see is filebeat and kibana.
Now after a lot of trying I get entries in kibana, except the actual message is missing.
This is the filebeat.yml
```
# filebeat.yml
filebeat.inputs:
- type: log
paths:
- '/usr/share/filebeat/dockerlogs/*/*.log.json'
json:
keys_under_root: true
add_error_key: true
message_key: message
ignore_decoding_error: true
output:
logstash:
hosts: ["logstash:5044"]
ssl.certificate_authorities: ["/usr/share/filebeat/cert/logstash_ca.crt"]
ssl.certificate: "/usr/share/filebeat/cert/logstash_beat_client.crt"
ssl.key: "/usr/share/filebeat/cert/logstash_beat_client.key"
logging.level: debug
logging.json: false
logging.metrics.enabled: false
ssl.verification_mode: none
```
And I can see that in both the logs and the temporary file output the message field is present.
Though I can not see them in kibana, the people who manage logstash have ensured me they do not filter on the beats input. This is an excerpt of the output file:
```
{
"@timestamp": "2019-06-25T09:04:49.719Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.3"
},
"host": {
"name": "10a58a8474b2"
},
"level": "INFO",
"@version": "1",
"message": "Starting AuthorizationServiceApplication on svanschooten-VirtualBox with PID 5111 (/home/svanschooten/Workspace/mn-aut/target/classes started by svanschooten in /home/svanschooten/Workspace/mn-aut)",
"logger_name": "com.mn.authorisation.AuthorizationServiceApplication",
"input": {
"type": "log"
},
"beat": {
"name": "10a58a8474b2",
"hostname": "10a58a8474b2",
"version": "6.5.3"
},
"level_value": 20000,
"source": "/usr/share/filebeat/dockerlogs/authorisation/mn-aut.log.json",
"offset": 620569,
"thread_name": "main",
"prospector": {
"type": "log"
}
}
```
Though I only see this in kibana:
```
{
"_index": "beats-2019.06.25",
"_type": "doc",
"_id": "QYHhjWsBPMQwkJWzluTq",
"_score": 1,
"_source": {
"log_origin": "beats_5044",
"level": "DEBUG",
"beat": {
"name": "10a58a8474b2",
"version": "6.5.3",
"hostname": "10a58a8474b2"
},
"origin": "filebeat",
"tags": [
"beats5044",
"beats_input_codec_plain_applied"
],
"offset": 620569,
"thread_name": "main",
"source": "/usr/share/filebeat/dockerlogs/authorisation/mn-aut.log.json",
"@version": "1",
"level_value": 10000,
"host": "10a58a8474b2",
"logger_name": "com.mn.authorisation.AuthorizationServiceApplication",
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"@timestamp": "2019-06-25T09:04:49.719Z"
},
"fields": {
"@timestamp": [
"2019-06-25T09:04:49.719Z"
]
}
}
```
When I use the 'Docker' input there are issues with parsing the lines but then I do get the log lines (though I have no control on which container logs are shipped). As we deploy multiple times a week it is unfeasible to update the container ID's every time, so this approach is not wanted.
---
### Author:  [@faec](https://discuss.elastic.co/u/faec)
#### Post date: [July 5, 2019, 8:02pm UTC](https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296/2 "2019-07-05T20:02:04Z")
Welcome! A few ideas about your configuration: first, you might want to remove the `json.message_key` as that should only affect configurations that use filtering and / or multiline settings and otherwise will just complicate things. For troubleshooting you might try turning off `keys_under_root` as well since it will give a clearer separation between the decoded fields and the rest, especially since `message` is a field name that is stripped by many common configurations.
Beyond that, is `message` stripped in all entries or only some? I notice in the example you've given the kibana appears to be a different doc than the output. Do you get results if you search for the presence of the field? E.g. in Kibana you would try:
```auto
GET /_search
{
"query": {
"exists": {
"field": "message"
}
}
}
```
Beyond that, it would be nice to see an example log line you're parsing this from to look for other possible issues. Good luck!
---
### Author:  [@svanschooten](https://discuss.elastic.co/u/svanschooten)
#### Post date: [July 23, 2019, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296/3 "2019-07-23T14:58:49Z")
Thanks for your advice, I will try to get it working properly.
But yeah, it is missing from all messages, even though the two examples I gave are different.
When I remove `message_key` and/or `keys_under_root` or even remove the whole json parsing section, the `message` field is present in the filebeat logs, but not received by kibana.
Querying whether there is a `message` field somewhere only yield results from other indices and projects... Though "remapping" the `message` field to something like `log_message` does yield results, so this is my workaround for now...
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 20, 2019, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-shipping-incomplete-logs/187296/4 "2019-08-20T14:58:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.