Multiline Java exceptions are set as non-existent

Elastic Stack Beats
1

I'm very new to ELK and I don't understand much of the internals , so I'm not sure if this is actually an issue with Kibana or Filebeat.

My setup is Filebeat -> Elastic -> Kibana v 6.2.2.

Whenever the "log" field of a document contains a multi-line Java exception, it is marked as non-existent and can't be searched by that field, even though the field contains the exception's text. In the following image I'm serching for documents from a service with a non-existent value for the field "log":

ELK
ELK.png1632×929 137 KB

The content actually exists, but for some reason Filebeat marked it as unexistent, or Kibana interprets it that way.

This is my filebeat.yml configuration:

#==========================  Modules configuration =============================
filebeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog*"]
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

#------------------------------- Auditd Module -------------------------------
- module: auditd
  log:
    enabled: false

#=========================== Filebeat prospectors ==============================
filebeat.prospectors:
- type: log
  enabled: true
  paths:
     - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.add_error_key: true
  json.keys_under_root: true
  processors:
    - add_docker_metadata: ~
    - add_cloud_metadata: ~
    - add_locale: ~
    - drop_event:
        when:
          or:
            - regexp:
                docker.container.labels.com.docker.swarm.service.name: ".*_kibana"
            - regexp:
                docker.container.labels.com.docker.swarm.service.name: ".*_cadvisor"
  multiline.pattern: "^\t|^[[:space:]]+(at|...)|^Caused by:"
  multiline.match: after

#========================== Elasticsearch output ===============================
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST}:9200"]
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

xpack.monitoring:
  enabled: true
  elasticsearch:

#============================== Dashboards =====================================
setup.dashboards:
  enabled: true
setup.kibana:
  host: "${KIBANA_HOST}:5601"
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
2

I don't understand the question, in particular what do you mean by "marked as non-existent"?

Your query looks to me like it should show all records in which the log field does not contain the word "exists". The log fields on the screen do not appear to contain the word "exists" so the output looks OK to me?

3

No, with that query Kibana is searching for documents that have a field log with an "existing" value.
ELK3

For some reason they use the same syntax for "exists" and "is" in the Kibana GUI, but they have a different meaning. This is the actual elasticsearch query from the Query DSL editor:

{
  "exists": {
    "field": "log"
  }
}

which is different from:

{
  "query": {
    "match": {
      "log": {
        "query": "exists",
        "type": "phrase"
      }
    }
  }
}

I'm actually filtering for documents in which the log field has no value, "NOT log: exists", and as you can see, documents with content in the log are shown.

My issue is that documents with multi-line Java exceptions behave as if they had no content in the log field. Documents with single line logs work perfectly. The only "multi-line" manipulation is happening with filebeat, so I'm not sure if maybe I miss-configured something.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.