Our team is working on creating some instant messaging applications.
We would like to gather logs from our applications with logstash.
As we are developing, we can use whatever technique to produce/store logs before logstash process them (we can write them to a file, store in a DB, pass over TCP etc.)
We can use different formats. It may be json, plain text, key-value pairs and so one.
What would be the best way to produce/store logs for subsequent processing in logstash to make the work for logstash as easy as possible?
I'd like to avoid using grok (because it CPU consuming), multiline filter (because I had a problem with it in the past, which still not resolved) and other CPU consuming plugins.
It'd be great if we can have some human-readable and logstash friendly format in one place, so that it is easy to read for support team and at the same time it is easy to process for logstash, but if there is now such option we can use several output options from our applications: one for human-support for in-place inspection and other for logstash.
Could you please recommend some log formats and backends?