How to Ship/Parse JSON logs (from Kibana) to Elasticsearch?

brolo · June 18, 2018, 4:46pm

I am trying to ship logs from a Kibana instance running in a docker container. I noticed that there is a module for Kibana in development (posted Github link below) and I pulled the pipeline code from Github and attempted to send the logs to this kibana pipeline in elastic search. It doesn't seem to parse the JSON fields at all; in Kibana the message field just contains the unparsed JSON log (seen below).

{"type":"response","@timestamp":"2018-06-18T15:25:47Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"user-agent":"check_http/v2.2 (monitoring-plugins 2.2)","connection":"close"},"remoteAddress":"xxx.xx.xx.xxx","userAgent":"xxx.xx.xx.xxx"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /api/status 200 2ms - 9.0B"}

I am using the autodiscover feature and this is my full filebeat config (v6.3.0).

setup.kibana:
  host: {{ourhoust}} #changed for privacy.

output.elasticsearch:
  hosts: ["somehost:port"]
  index: "filebeats"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644

filebeat.modules:
  - module: system
  - module: nginx

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains.docker.container.image: "nginx"
          config:
            - module: nginx
              access:
                input:
                  type: docker
                  containers.ids: "${data.docker.container.id}"
                  containers.stream: stdout
              error:
                input:
                  type: docker
                  containers.ids: "${data.docker.container.id}"
                  containers.stream: stderr
        - condition:
            contains.docker.container.image: "docker.elastic.co/kibana/kibana"
          config:
            - type: docker
              containers.ids: "${data.docker.container.id}"
              input:
                type: log
                pipeline: kibana-module-copy
                #json.keys_under_root: true
                #json.add_error_key: true
                #json.message_key: log

andrewkroh · June 19, 2018, 2:57pm

It sounds like Filebeat was not configured to parse the JSON logs generated by Kibana (see config from module). The ingest pipeline from the Kibana filebeat module does not do any JSON parsing and expects it to already be parsed. Because of this there should be an error.message field added to each event by the ingest pipeline. Do you see that error? Are you sure the data is being passed to the pipeline?

You can see stats from the ingest pipelines by querying GET _nodes/stats/ingest and looking for kibana-module-copy. It will tell you the number of events and number of failures.

brolo · June 22, 2018, 12:13pm

Ok so I found out that my old filebeat.yml config wasn't correct, to specify a pipeline while using autodiscover it needs to be done directly under the config: field. My new filebeat.yml config is below.

setup.kibana:
  host: {{ourhoust}} #changed for privacy.

output.elasticsearch:
  hosts: ["somehost:port"]
  index: "filebeats"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644

filebeat.modules:
  - module: system
  - module: nginx

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains.docker.container.image: "nginx"
          config:
            - module: nginx
              access:
                input:
                  type: docker
                  containers.ids: "${data.docker.container.id}"
                  containers.stream: stdout
              error:
                input:
                  type: docker
                  containers.ids: "${data.docker.container.id}"
                  containers.stream: stderr
        - condition:
            contains.docker.container.image: "docker.elastic.co/kibana/kibana"
          config:
            - type: docker
              containers.ids: "${data.docker.container.id}"
              pipeline: kibana-module-copy
              input:
                json.keys_under_root: false
                json.add_error_key: true

After these changes, querying GET _nodes/stats/ingest shows that the pipeline is receiving the logs from Kibana, however, they are still not being parsed. I am receiving this error message: field [json] not present as part of path [json].

Not really sure how to fix that, since the test logs for the Kibana module in development are the same format as my current logs (and the test logs seem to be parsed correctly from what I can see in GitHub). Any Ideas?

steffens · June 22, 2018, 2:05pm

You kibana docker input must be configured more like this:

- type: docker
  containers.ids: "${data.docker.container.id}"
  pipeline: kibana-module-copy
  json.keys_under_root: false
  json.add_error_key: true

The type: docker indicates it is already an input -> remove the input namespace.
The input config namespace is specific to modules (your nginx autodiscovery is indeed using filebeat modules). In filebeat modules provide additional abstractions and configurations on top of inputs in filebeat and other components in the Elastic Stack.

brolo · June 22, 2018, 2:48pm

So before I made the changes that you had suggested I checked the GET -nodes/stats/ingest info for my pipeline. This was the output:

"kibana-module-copy": {
            "count": 1622,
            "time_in_millis": 0,
            "current": 0,
            "failed": 0
}

After making the changes that you suggested, that same query now returns a count of 1622 (like the pipeline is no longer receiving logs).

Checking Kibana, I no longer see any logs at all. Before the change I was seeing the above mentioned error message.

Current filebeat config (only the relevant changed section):

        - condition:
            contains.docker.container.image: "docker.elastic.co/kibana/kibana"
          config:
            - type: docker  
              containers.ids: "${data.docker.container.id}"
              pipeline: kibana-module-copy
              json.keys_under_root: false
              json.add_error_key: true

steffens · June 22, 2018, 3:25pm

So, no more kibana related events in Elasticsearch?

Have you checked the filebeat registry offset? Did filebeat reach the end of the log file by any chance?

As the ingest pipeline runs in Elasticsearch, have you checked Elasticsearch logs?

brolo · June 22, 2018, 3:51pm

I think I have tracked down the issue. I needed to update Kibana to version 6.3.0. I am receiving logs now and the pipeline seems to be partially parsing the log, however, I am now running into an error message saying
Unexpected character ('r' (code 114)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@37c5d5c3; line: 1, column: 3]

and similar errors referencing different characters.

After some more analysis, it seems that all of the fields are being parsed, but none of the fields are renamed as defined in the pipeline (I see the native JSON fields in Kibana). Not exactly sure why there are errors in every entry for the Kibana log still.

Edit:
Fixed the above issue. My pipeline was incorrectly configured. Full pipeline below.

{
    "description": "Pipeline for parsing Kibana logs",
    "on_failure": [
        {
            "set": {
                "field": "error.message",
                "value": "{{ _ingest.on_failure_message }}"
            }
        }
    ],
    "processors": [
        {
            "rename": {
                "field": "json",
                "target_field": "kibana.log.meta"
            }
        },
        {
            "rename": {
                "field": "kibana.log.meta.@timestamp",
                "target_field": "read_timestamp"
            }
        },
        {
            "rename": {
                "field": "kibana.log.meta.message",
                "target_field": "message"
            }
        },
        {
            "rename": {
                "field": "kibana.log.meta.state",
                "target_field": "kibana.log.state",
                "ignore_missing": true
            }
        },
        {
            "rename": {
                "field": "kibana.log.meta.pid",
                "target_field": "process.pid"
            }
        },
        {
            "rename": {
                "field": "kibana.log.meta.tags",
                "target_field": "kibana.log.tags"
            }
        },
        {
            "date": {
                "field": "read_timestamp",
                "formats" : ["ISO8601"],
                "target_field": "@timestamp"
            }
        },
        {
            "append": {
                "field": "service.name",
                "value": "kibana"
            }
        }
    ]
}

system · July 20, 2018, 3:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.