### Author:  [@brolo](https://discuss.elastic.co/u/brolo)
#### Post date: [June 18, 2018, 4:46pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/1 "2018-06-18T16:46:41Z")
I am trying to ship logs from a Kibana instance running in a docker container. I noticed that there is a module for Kibana in development (posted Github link below) and I pulled the pipeline code from Github and attempted to send the logs to this kibana pipeline in elastic search. It doesn't seem to parse the JSON fields at all; in Kibana the message field just contains the unparsed JSON log (seen below).
>
`{"type":"response","@timestamp":"2018-06-18T15:25:47Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"user-agent":"check_http/v2.2 (monitoring-plugins 2.2)","connection":"close"},"remoteAddress":"xxx.xx.xx.xxx","userAgent":"xxx.xx.xx.xxx"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /api/status 200 2ms - 9.0B"}`
I am using the autodiscover feature and this is my full filebeat config (v6.3.0).
```auto
setup.kibana:
host: {{ourhoust}} #changed for privacy.
output.elasticsearch:
hosts: ["somehost:port"]
index: "filebeats"
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat.log
keepfiles: 7
permissions: 0644
filebeat.modules:
- module: system
- module: nginx
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
contains.docker.container.image: "nginx"
config:
- module: nginx
access:
input:
type: docker
containers.ids: "${data.docker.container.id}"
containers.stream: stdout
error:
input:
type: docker
containers.ids: "${data.docker.container.id}"
containers.stream: stderr
- condition:
contains.docker.container.image: "docker.elastic.co/kibana/kibana"
config:
- type: docker
containers.ids: "${data.docker.container.id}"
input:
type: log
pipeline: kibana-module-copy
#json.keys_under_root: true
#json.add_error_key: true
#json.message_key: log
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [June 19, 2018, 2:57pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/2 "2018-06-19T14:57:21Z")
> [@brolo](#):
>
> the message field just contains the unparsed JSON log
It sounds like Filebeat was not configured to parse the JSON logs generated by Kibana (see [config](https://github.com/elastic/beats/blob/56d280df170ade258e98a3d59187b19a89fc0df2/filebeat/module/kibana/log/config/log.yml#L8-L9) from module). The ingest pipeline from the Kibana filebeat module does not do any JSON parsing and expects it to already be parsed. Because of this there should be an `error.message` field added to each event by the ingest pipeline. Do you see that error? Are you sure the data is being passed to the pipeline?
You can see stats from the ingest pipelines by querying `GET _nodes/stats/ingest` and looking for `kibana-module-copy`. It will tell you the number of events and number of failures.
---
### Author:  [@brolo](https://discuss.elastic.co/u/brolo)
#### Post date: [June 22, 2018, 12:13pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/3 "2018-06-22T12:13:58Z")
Ok so I found out that my old `filebeat.yml` config wasn't correct, to specify a pipeline while using autodiscover it needs to be done directly under the `config:` field. My new `filebeat.yml` config is below.
```auto
setup.kibana:
host: {{ourhoust}} #changed for privacy.
output.elasticsearch:
hosts: ["somehost:port"]
index: "filebeats"
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat.log
keepfiles: 7
permissions: 0644
filebeat.modules:
- module: system
- module: nginx
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
contains.docker.container.image: "nginx"
config:
- module: nginx
access:
input:
type: docker
containers.ids: "${data.docker.container.id}"
containers.stream: stdout
error:
input:
type: docker
containers.ids: "${data.docker.container.id}"
containers.stream: stderr
- condition:
contains.docker.container.image: "docker.elastic.co/kibana/kibana"
config:
- type: docker
containers.ids: "${data.docker.container.id}"
pipeline: kibana-module-copy
input:
json.keys_under_root: false
json.add_error_key: true
```
After these changes, querying `GET _nodes/stats/ingest` shows that the pipeline is receiving the logs from Kibana, however, they are still not being parsed. I am receiving this error message: `field [json] not present as part of path [json]`.
Not really sure how to fix that, since the test logs for the Kibana module in development are the same format as my current logs (and the test logs seem to be parsed correctly from what I can see in GitHub). Any Ideas?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 22, 2018, 2:05pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/4 "2018-06-22T14:05:14Z")
You kibana docker input must be configured more like this:
```auto
- type: docker
containers.ids: "${data.docker.container.id}"
pipeline: kibana-module-copy
json.keys_under_root: false
json.add_error_key: true
```
The `type: docker` indicates it is already an input -\> remove the `input` namespace.
The `input` config namespace is specific to modules (your nginx autodiscovery is indeed using filebeat modules). In filebeat modules provide additional abstractions and configurations on top of inputs in filebeat and other components in the Elastic Stack.
---
### Author:  [@brolo](https://discuss.elastic.co/u/brolo)
#### Post date: [June 22, 2018, 2:48pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/5 "2018-06-22T14:48:31Z")
So before I made the changes that you had suggested I checked the `GET -nodes/stats/ingest` info for my pipeline. This was the output:
```auto
"kibana-module-copy": {
"count": 1622,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
```
After making the changes that you suggested, that same query now returns a count of 1622 (like the pipeline is no longer receiving logs).
Checking Kibana, I no longer see any logs at all. Before the change I was seeing the above mentioned error message.
Current filebeat config (only the relevant changed section):
```auto
- condition:
contains.docker.container.image: "docker.elastic.co/kibana/kibana"
config:
- type: docker
containers.ids: "${data.docker.container.id}"
pipeline: kibana-module-copy
json.keys_under_root: false
json.add_error_key: true
```
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 22, 2018, 3:25pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/6 "2018-06-22T15:25:45Z")
So, no more kibana related events in Elasticsearch?
Have you checked the filebeat registry offset? Did filebeat reach the end of the log file by any chance?
As the ingest pipeline runs in Elasticsearch, have you checked Elasticsearch logs?
---
### Author:  [@brolo](https://discuss.elastic.co/u/brolo)
#### Post date: [June 22, 2018, 3:51pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/7 "2018-06-22T15:51:54Z")
I think I have tracked down the issue. I needed to update Kibana to version 6.3.0. I am receiving logs now and the pipeline seems to be partially parsing the log, however, I am now running into an error message saying
`Unexpected character ('r' (code 114)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@37c5d5c3; line: 1, column: 3]`
and similar errors referencing different characters.
After some more analysis, it seems that all of the fields are being parsed, but none of the fields are renamed as defined in the pipeline (I see the native JSON fields in Kibana). Not exactly sure why there are errors in every entry for the Kibana log still.
Edit:
Fixed the above issue. My pipeline was incorrectly configured. Full pipeline below.
```auto
{
"description": "Pipeline for parsing Kibana logs",
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
],
"processors": [
{
"rename": {
"field": "json",
"target_field": "kibana.log.meta"
}
},
{
"rename": {
"field": "kibana.log.meta.@timestamp",
"target_field": "read_timestamp"
}
},
{
"rename": {
"field": "kibana.log.meta.message",
"target_field": "message"
}
},
{
"rename": {
"field": "kibana.log.meta.state",
"target_field": "kibana.log.state",
"ignore_missing": true
}
},
{
"rename": {
"field": "kibana.log.meta.pid",
"target_field": "process.pid"
}
},
{
"rename": {
"field": "kibana.log.meta.tags",
"target_field": "kibana.log.tags"
}
},
{
"date": {
"field": "read_timestamp",
"formats" : ["ISO8601"],
"target_field": "@timestamp"
}
},
{
"append": {
"field": "service.name",
"value": "kibana"
}
}
]
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 20, 2018, 3:51pm UTC](https://discuss.elastic.co/t/how-to-ship-parse-json-logs-from-kibana-to-elasticsearch/136345/8 "2018-07-20T15:51:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.