Using Logstash to analyse log4j log files


Can you give me a basic example on how to configure logstash to handle log4j files as input including how to give an entire directory to look into.

Thank you.

Log files produced by Log4j can have just about any format so specific advice is next to impossible. Use a file input to read the files and a grok filter to parse them. The Logstash documentation contains examples of how to read httpd log files and the same general principles apply in your case. I suppose log messages can span over multiple lines (like e.g. Java stacktraces) so you'll probably want to use a multiline codec, but you can add that later.

Common pitfalls when dealing with a directory full of (presumably old) files:

  • By default recent versions of Logstash ignores files older than 24 hours. Use the file input's ignore_older option to change this.
  • Make sure you understand sincedb and the file input's start_position option if you want Logstash to read existing files from the beginning.

Thank you, it seems to be useful.

Another option is to use the log4j input. Define a socket appender in your log4j properties file for your logstash server and an open port. The input automatically creates fields based on log4J.LoggingEvent fields.

Hi Craig,

Thank you, but can you give me a simple basic example? It will be very useful.

Thank you in advance.

On the logstash side, the log4j input is pretty simple:

        port => ####

The server was already set up to do log4j logging so it was just a matter of adding a log4J SocketAppender:

#add the socketappender by name to the rootlogger

#configure socketappender
log4j.appender.logstash.RemoteHost=[your logstash server hostname]

I believe the log4j.LoggingEvent object is serialized and sent to logstash where it's fields are used to create JSON fields, i.e. message, level, etc. No need for a filter, just output to elasticsearch.