Multiline Java exceptions are set as non-existent

[AndresPineros (Andrés Felipe Piñeros)]

I'm very new to ELK and I don't understand much of the internals , so I'm not sure if this is actually an issue with Kibana or Filebeat.

My setup is Filebeat -> Elastic -> Kibana v 6.2.2.

Whenever the "log" field of a document contains a multi-line Java exception, it is marked as non-existent and can't be searched by that field, even though the field contains the exception's text. In the following image I'm serching for documents from a service with a non-existent value for the field "log":

The content actually exists, but for some reason Filebeat marked it as unexistent, or Kibana interprets it that way.

This is my filebeat.yml configuration:

#==========================  Modules configuration =============================
filebeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog*"]
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

#------------------------------- Auditd Module -------------------------------
- module: auditd
  log:
    enabled: false

#=========================== Filebeat prospectors ==============================
filebeat.prospectors:
- type: log
  enabled: true
  paths:
     - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.add_error_key: true
  json.keys_under_root: true
  processors:
    - add_docker_metadata: ~
    - add_cloud_metadata: ~
    - add_locale: ~
    - drop_event:
        when:
          or:
            - regexp:
                docker.container.labels.com.docker.swarm.service.name: ".*_kibana"
            - regexp:
                docker.container.labels.com.docker.swarm.service.name: ".*_cadvisor"
  multiline.pattern: "^\t|^[[:space:]]+(at|...)|^Caused by:"
  multiline.match: after

#========================== Elasticsearch output ===============================
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST}:9200"]
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

xpack.monitoring:
  enabled: true
  elasticsearch:

#============================== Dashboards =====================================
setup.dashboards:
  enabled: true
setup.kibana:
  host: "${KIBANA_HOST}:5601"
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

[TimWard]

I don't understand the question, in particular what do you mean by "marked as non-existent"?

Your query looks to me like it should show all records in which the log field does not contain the word "exists". The log fields on the screen do not appear to contain the word "exists" so the output looks OK to me?

[AndresPineros (Andrés Felipe Piñeros)]

No, with that query Kibana is searching for documents that have a field log with an "existing" value.

For some reason they use the same syntax for "exists" and "is" in the Kibana GUI, but they have a different meaning. This is the actual elasticsearch query from the Query DSL editor:

{
  "exists": {
    "field": "log"
  }
}

which is different from:

{
  "query": {
    "match": {
      "log": {
        "query": "exists",
        "type": "phrase"
      }
    }
  }
}

I'm actually filtering for documents in which the log field has no value, "NOT log: exists", and as you can see, documents with content in the log are shown.

My issue is that documents with multi-line Java exceptions behave as if they had no content in the log field. Documents with single line logs work perfectly. The only "multi-line" manipulation is happening with filebeat, so I'm not sure if maybe I miss-configured something.