Using Logstash to analyse log4j log files

Elastic Stack Logstash
1

Hi,

Can you give me a basic example on how to configure logstash to handle log4j files as input including how to give an entire directory to look into.

Thank you.

2

Log files produced by Log4j can have just about any format so specific advice is next to impossible. Use a file input to read the files and a grok filter to parse them. The Logstash documentation contains examples of how to read httpd log files and the same general principles apply in your case. I suppose log messages can span over multiple lines (like e.g. Java stacktraces) so you'll probably want to use a multiline codec, but you can add that later.

Common pitfalls when dealing with a directory full of (presumably old) files:

  • By default recent versions of Logstash ignores files older than 24 hours. Use the file input's ignore_older option to change this.
  • Make sure you understand sincedb and the file input's start_position option if you want Logstash to read existing files from the beginning.
3

Thank you, it seems to be useful.

4

Another option is to use the log4j input. Define a socket appender in your log4j properties file for your logstash server and an open port. The input automatically creates fields based on log4J.LoggingEvent fields.

5

Hi Craig,

Thank you, but can you give me a simple basic example? It will be very useful.

Thank you in advance.

6

On the logstash side, the log4j input is pretty simple:

input{
    log4j{
        port => ####
    }
}

The server was already set up to do log4j logging so it was just a matter of adding a log4J SocketAppender:

#add the socketappender by name to the rootlogger
log4j.rootlogger=DEBUG,file,logstash

#configure socketappender
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.Port=####
log4j.appender.logstash.RemoteHost=[your logstash server hostname]
log4j.appender.logstash.ReconnectionDelay=10000

I believe the log4j.LoggingEvent object is serialized and sent to logstash where it's fields are used to create JSON fields, i.e. message, level, etc. No need for a filter, just output to elasticsearch.