After taking a step back I notice a lot of messages are useless. And I want to add a tag on them to easily filter while on Kibana. The message is actually logs that span about 70 lines that when put together describe a health/ping message. And this happens every 1-2 seconds so its important that I figure some way to easily identify and filter it out. Here is a quick snip:
20150819 14:15:03.800[INFO ]- Invoking HTTP method POST
20150819 14:15:03.801[INFO ]- URIMappingInterceptor can only handle HTTP GET, not HTTP POST
...
20150819 14:15:03.900[INFO ]- Interceptors contributed by service: []
20150819 14:15:03.908[INFO ]- Outbound Message
What are the characteristics of this burst of messages? Do they always start with "Invoking HTTP method POST" and end with "Outbound Message"? Are those messages ever interleaved with useful messages that you want to keep?
I thought the invoking http method post was it but actually the start of these health messages is this:
20150810 16:24:21.043[INFO ]-Inbound Message
----------------------------
ID: 1159
...
--------------------------------------
ending with
20150810 16:24:24.057[INFO ]-Outbound Message
---------------------------
ID: 1159
...
--------------------------------------
It seems like all calls have this same basic structure and in between these messages can vary from 10-200+ lines. So now my question what is the best method to group these chunks of messages? I am thinking right now I group the inbound and outbound messages already as they are shipped from the host. Now when they get to logstash maybe I multiline them such that if it doesn't begin with pattern with timestamp -\w+ Message then append to previous. ( or maybe do this on log forwarder).
No these messages are not interleaved with useful messages.
Which brings to mind is an ELK stack really optimized for this use case? From this one log source my messages are turning into these gigantic messages of about 500+ lines at the worst case. But normally they should just reach 100-200 lines.
Now I am trying to turn multi-line logs into readable messages that do not overwhelm people at first glance.