I have an ELK setup which uses the gelf input plugin.
Currently, logs are sent from Node.js using Bunyan and the dependency gelf-stream, which in term depends on gelfling. The code for creating a message is here: https://github.com/mhart/gelf-stream/blob/master/gelf-stream.js#L83.
Messages arrive in logstash with short_message, which gets passed through to elasticsearch and kibana.
I'm working on switching to a new logging system using log4js and their gelf appender. The code is very similar: https://github.com/log4js-node/gelf/blob/master/lib/index.js#L92
But these messages appear without short_message attached. I've raised this as an issue on the repo https://github.com/log4js-node/gelf/issues/5, but I'm trying to debug further using the logstash logs.
I have temporarily added stdout to my output config and I've also increased the log levels:
curl -GET 'localhost:9600/_node/logging?pretty''
{
...
"loggers" : {
"logstash.agent" : "DEBUG",
...
"logstash.inputs.gelf" : "TRACE",
"logstash.inputs.metrics" : "DEBUG",
...
}
I've tried both TRACE and DEBUG, but the logstash.inputs.gelf plugin doesn't seem to output any debug info. For each message received, inputs.metrics adds debug info, but there's no other debug info.
Adding stdout to the outputs is fine, but it doesn't prove that the message arrives without short_message - I was hoping the debug output would show this.
Note: I've already banged my head against the requirement that gelf messages be gzipped. The gelf input plugin seems a bit unloved.
Questions:
- Is it known and expected that the gelf input plugin is missing debug info?
- Anyone have any idea about any subtle rules of the gelf input plugin that might cause short_message to disappear?
- Is it unwise to use the gelf input plugin? i.e. is it not really a first class citizen? If not what alternatives should/could I use - i.e. would logstash udp be better?