Multiline java exceptions cannot be searchable in Kibana

[himaz.m (Himaz)]

I'm using filebeat + logstash + elasticsearch + kibana version 5.0

I have a log file with following format;

2016-11-20 06:12:54 | [ajp-nio-8009-exec-2195] | ADMIN_API | INFO | c.i.admin.api.web.FileController - login to the system | {"logData":{"span":"asdf1", "action":"login", "action_stage":"login endpoint"}} | 
2016-11-20 06:12:54 | [ajp-nio-8009-exec-2195] |  | INFO | c.i.admin.api.web.FileController - login to the system |  | 
2016-11-21 07:12:54 | [ajp-nio-8009-exec-2195] | ADMIN_API | ERROR | c.i.admin.api.web.FileController - login to the system |  | 
com.incentivio.exutil.NullPointerException: null
	at com.incentivio.order.domain.item.ItemService.getItemById(ItemService.java:31)
	at com.incentivio.order.application.orderitem.OrderItemAppService.populateOrderItems(OrderItemAppService.java:173)
	at com.incentivio.order.application.orderitem.OrderItemAppService.newAddOrderItem(OrderItemAppService.java:87)
....

My logstash input config

input {
 beats {
  port => 5044
  codec => multiline {
   pattern => "(^%{TIMESTAMP_ISO8601})"
   negate => true
   what => "previous"
  }
 }
}

My logstash grok filter

filter {
  if [type] == "mixlog" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \| \[%{DATA:thread}\] \| %{DATA:module} \| %{DATA:loglevel} \| %{JAVACLASS:class} - %{GREEDYDATA:message} \| %{GREEDYDATA:jsonbody} \| %{GREEDYDATA:exception}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
      overwrite => [ "message" ]
    }
 json {
          source => "jsonbody"
    }
    date {
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
    }
    mutate {
      remove_field => [ "_score", "_type", "timestamp", "jsonbody", "@version", "offset" ]
    }
  }
}

I can search using almost all the fields (@timestamp, loglevel, message, etc) in Kibana except 'exception'. When I try to search for strings like null or NullPointerException I get No results found message in Kibana.

Json doc I receive in Kiabana

{
  "_index": "filebeat-2016.11.21",
  "_type": "mixlog",
  "_id": "AViwEBXVaLpKvXLDjIkf",
  "_score": null,
  "_source": {
    "exception": "\ncom.incentivio.exutil.NullPointerException: null\n\tat com.incentivio.order.domain.item.ItemService.getItemById(ItemService.java:31)\n\tat com.incentivio.order.application.orderitem.OrderItemAppService.populateOrderItems(OrderItemAppService.java:173)\n\tat com.incentivio.order.application.orderitem.OrderItemAppService.newAddOrderItem(OrderItemAppService.java:87)\n\tat sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)",
    "module": "ADMIN_API",
    "input_type": "log",
    "source": "/opt/MIXLOG/fixed_pipe.log",
    "thread": "ajp-nio-8009-exec-2195",
    "message": "login to the system",
    "type": "mixlog",
    "tags": [
      "multiline",
      "beats_input_codec_multiline_applied"
    ],
    "received_from": "fbeat.aeturnum.com",
    "@timestamp": "2016-11-21T07:12:54.000Z",
    "received_at": "2016-11-29T12:29:33.699Z",
    "loglevel": "ERROR",
    "beat": {
      "hostname": "fbeat.aeturnum.com",
      "name": "fbeat.aeturnum.com",
      "version": "5.0.0"
    },
    "host": "fbeat.aeturnum.com",
    "class": "c.i.admin.api.web.FileController"
  },
  "fields": {
    "received_at": [
      1480422573699
    ],
    "@timestamp": [
      1479712374000
    ]
  },
  "sort": [
    1479712374000
  ]
}

Any idea on how to search for a partial text from the exception field?

[eperry (Ed)]

Simple question did you refresh the index in kibana . I know in K4 you need to refresh the index under settings to see the new fields. (It normaly has a yellow ! next to the field that it is not searchable yet)

Sorry, I have not played with 5 but I suspect it is there.

Btw, can you issue a "curl" search on the field that would at least narrow down the issue to kibana.

[himaz.m (Himaz)]

I didn't add the exception field recently, I added it with other fields. Yes, I tried refreshing the indexes, but still the same.

[eperry (Ed)]

what is the "query" that you are issuing?

[himaz.m (Himaz)]

When I enter the string login it gives me the following output.

But when I enter null it says No results found as below

But in the 1st image you can see the exception field contains a string null. But when I particularly search for it, it doesn't appear. Is it because the exception is a multiline string? If so how can I overcome to resolve this?

[himaz.m (Himaz)]

Do I need to explicitly create any index or template file for this?

[himaz.m (Himaz)]

I noticed the type of exception was keyword and was set to "ignore_above" : 1024". It worked after I added the below template.

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates" : [{
		"strings_as_keyword" : {
			"mapping" : {
				"ignore_above" : 1024,
				"index" : "not_analyzed",
				"type" : "string"
			},
			"match_mapping_type" : "string"
			}
		}],
      "properties": {
        "timestamp": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" },
        "thread": { "type": "keyword", "ignore_above" : 1024 },
        "processid": { "type": "integer" },
		"module": { "type": "keyword", "ignore_above" : 1024 },
        "trace": { "type": "keyword", "ignore_above" : 1024 },
		"span": { "type": "keyword", "ignore_above" : 1024 },
        "loglevel": { "type": "keyword", "ignore_above" : 1024 },
		"class": { "type": "keyword", "ignore_above" : 1024 },
		"message": { "type": "text", "index": "analyzed" },
		"exception": { "type": "text", "index": "analyzed" }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

[eperry (Ed)]

excelent, glad you figured it out

[sjivan (Sanjiv Jivan)]

I experienced the same issue. Great find, thanks!

[himaz.m (Himaz)]

Thank you all