# Filebeat fails to process kibana json logs “failed to format message from \*json-.log “in a docker enviroment with logstash
**URL:** https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295
**Category:** Logstash
**Tags:** docker
**Created:** [July 9, 2021, 4:54pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295 "2021-07-09T16:54:29Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Annette1](https://discuss.elastic.co/u/Annette1)
#### Post date: [July 9, 2021, 4:54pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/1 "2021-07-09T16:54:29Z")
Problem description - since I have installed logstash I am seeing the following in kibana logs -
failed to format message from /var/lib/docker/.containers/xxx-json.log
If I remove logstash and send directly through elasticsearch I do not see these errors in kibana logs. It is also important to not that the data is getting to kibana. For some reason I am seeing error on some of the output. I will provide a sample at the end of this problem description.
filebeat config: (filebeat 6.8.6)
filebeat:
[{"paths": ["/var/log/docker/containers/_/_.log\*"], "fields": "paths": ["environment": "dev", "system" "test", "level": "docker-service"}, "json.keys\_under\_root": false, "tags": ["docker", "json", "dev"], "multiline": ["negate": true, "pattern": "^\[|^[0-9]{4]-[0-9]{2}-[0-9]{2}", "match": "after"}, "type": "log"} }]
config.modules:
path: ${path.config}/modules.d/\*.yml
reload.enabled: false
output:
logstash:
hosts: ["dev.logstash1.test:5044, "dev.logstash2.test:5044"]
loadbalance: true
logstash.config (7.7.0)
input {
beats =\> 5044
host =\> "x.x.x.x"
}
}
output {
elasticsearch {
hosts=\> ["[https://dev.logs1.test:9200](https://dev.logs1.test:9200)", "[https://dev.logs2.test:9200](https://dev.logs2.test:9200)"]
ssl\_certificate\_verification =\> false
index =\> "logstash-app-%{+YYYY.MM.dd}"
}
}
}
elasticsearch config
cluster.name logger
node:
name: node1
master: true
data: true
ingest: true
path:
data:/usr/localshare/elasticsearch/applications/elasticsearch/data
logs: /usr/local/share/applications/elasticsearch/logs
bootstrap.memory\_lock: true
network:
host: localhost
tcp\_keep\_alive: true
http:
port: 9201
publish\_port: 9200
transport:
host: localhost
tcp.port: 9301
publish\_host: localhost
publish\_port: 19301
discovery.zen
minimum\_master\_nodes: 2
ping.unicast.hosts:
- localhost:19301
- localhost: 19302
- localhost: 19303
Kibana - (6.8.2)
kibana config
server.port: 5601
server.name "kibana"
elasticsearch.url: "[https://dev.logs1.test:9200](https://dev.logs1.test:9200)"
kibana.index: ".kibana"
kibana.defaultApppld: "discover"
```
server.ssl.enabled: true
server.ssl.certificate" "path_to_cert/cert.pem"
server.ssl.key: "path_to_cert/key.pem"
server.ssl.supportedProtocols: [TLSv1.2"]
elasticsearch.ssl.certificate: "path_to_cert/cert.pem"
elasticsearch.ssl.key: "path_to_cert/key.pem"
elastisearch.ssl.vertificationMode: none
```
pid.file: "path\_to\_pid/kibana.pid"
logging.dest: "/path\_to\_log/kibana.log"
have tried with and without- the following with not success -
xpack.infra.sources.fields.massage: ['message', '@message', 'json.message']
xpack.monitoring.ui.container.elasticsearch.enabled: true
logging.silent: true
Sample data failing -
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 5:26pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/2 "2021-07-09T17:26:35Z")
I am guessing here, since I do not run filebeat, elasticsearch, or kibana, but ... my understanding of filebeat [modules](https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-modules-overview.html) is that they provide a way to parse a bunch of standard log file formats. filebeat does not do the parsing, instead it uses an ingest pipeline in elasticsearch. If you are sending data to logstash, then to use an ingest pipeline you would have to set the [pipeline](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-pipeline) option on the elasticsearch output.
---
### Author:  [@Annette1](https://discuss.elastic.co/u/Annette1)
#### Post date: [July 9, 2021, 6:37pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/3 "2021-07-09T18:37:04Z")
Thank-you for responding.
I think I am understanding your input here
On the logstash.conf, on the outputs, I am not pointing to elasticsearch, the next point in the path as show below.
data flow current filebeat =\> logstash =\>. elasticearch =\> kibana
data flow when problem not occurring - filebeat =\> elasticsearch =\> kibana
It is also important to note that the data is getting to kibana, as I can see output in discovery and the dashboards. I just get these errors on some the of the messages.json docker input (failed to format massage.....as referenced before...) It is also important to not that the reason I am using logstash is that I can also route to the s3 bucket as well as to kibana for online viewing. It is just since I have inserted logstah into the mix, that I am getting these additional messages/error messages.
Here is the output pipeline defined in logstash to elastisearch
output {
elasticsearch {
hosts=\> ["[https://dev.logs1.test:9200](https://dev.logs1.test:9200)";, "[https://dev.logs2.test:9200](https://dev.logs2.test:9200)";]
ssl\_certificate\_verification =\> false
index =\> "logstash-app-%{+YYYY.MM.dd}"
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/4 "2021-07-09T18:53:20Z")
When filebeat sends data directly to elasticsearch, in addition to the log file entries it sends metadata saying what format the log files are in. So if filebeat says "this is an IIS access log" then elasticsearch will process it using [this](https://github.com/elastic/beats/blob/master/filebeat/module/iis/access/ingest/pipeline.yml) ingest pipeline. If that processing does not happen then kibana will display the "failed to format message" error.
In logstatsh, that processing will not happen unless you set the pipeline option when sending data to elasticsearch. If you have multiple log formats you may need to configure filebeat to add a field that indicates which pipeline the event should be sent through.
ETA: It looks like filebeat adds the metadata for you. See the [documentation](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#CO21-1).
---
### Author:  [@Annette1](https://discuss.elastic.co/u/Annette1)
#### Post date: [July 9, 2021, 10:12pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/5 "2021-07-09T22:12:28Z")
Thanks again for responding.....
here is what I tried....almost verbatim from the documentation....
output {
if [@metadata][pipeline] {
elasticsearch {
hosts =\> "original hosts"
index =\> "original index"
pipeline =\> "%{[@metadata][pipeline]}"
}
} else {
elasticsearch {
hosts =\> "[https://original](https://original) hosts"
index =\> "original index "
if I just use the first part (if) ..nothing comes out.....
If I add in the second part( i.e. else).....the error comes back.....is there a step I missed?
Do I need to do this step from the doc?
filebeat setup --pipelines --modules nginx,system (perhaps for logstash?)
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 11:28pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/6 "2021-07-09T23:28:01Z")
Configure
```
output { stdout { codec => rubydebug { metadata => true } } }
```
and see if you can spot the name of the pipeline in an event.
---
### Author:  [@Annette1](https://discuss.elastic.co/u/Annette1)
#### Post date: [July 13, 2021, 12:15pm UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/7 "2021-07-13T12:15:17Z")
ok..figured out elasticsearch pipeline. I ran curl -k -v -X GET "[https://dev.logs1.test:9200/\_ingest\_pipeline/\*](https://dev.logs1.test:9200/_ingest_pipeline/*)" -H 'content-Type: application/json'
it returned back xpack\_monitoring\_2....put that in the pipeline with no joy.
---
### Author:  [@Annette1](https://discuss.elastic.co/u/Annette1)
#### Post date: [July 14, 2021, 11:40am UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/8 "2021-07-14T11:40:32Z")
ok..finally fixed the issue. The problem was that elasticsearch puts everything into a message file, including json.log data. Logstash puts json.log data into a json.log file. I updated the kibana.xml file parameter xpack.infra.default.fields.message: ['@message'. 'json.message', 'json.message', 'json.log'] with json.log being the key value and then the kibana.log no longer showed the failed to format message error and actually show the output of the log.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 11, 2021, 11:41am UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295/9 "2021-08-11T11:41:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.