Monitoring changes to application property files

Elastic Stack Logstash
1

We have several important application property files that we would like to monitor with Logstash (app1.properties, app2.properties, etc). Ideally, whenever anything changes to one of these application property files, we'd like to completely re-create the properties/app1 index/type pair.

In my experience, logstash (1.5.3 Win) file input plugin does a good job monitoring data that we append to the end of the file, but it doesn't recognize updates to any of the records in any other place in the file.

How can we solution the desired behavior in Logstash?!

2

Is there a way to capture changes to individual property items anywhere inside in the properties file using Logstash?!

3

Please.. help me out here... I'm out of ideas and need some help.

4

Logstash doesn't ship with a standard plugin for monitoring files in that way. I suppose you could use an exec input to pick up the contents of the property file, but I don't see how it would be possible to detect when it's changed.

5

I thought the intended purpose of the "start_position=beginning" setting was to force Logstash to read the content of the file top-to-bottom each time the file is changed, but I just re-read the doc and see that I was mistaken.

6

James, I saw your post and had a suggestion. I ran a across a tool called ScriptRock which may be able to help. Have you looked at it yet???

7

Maybe you could use exec input to type your file to stdout like :

exec {
    command => "type E:\myFile"
    interval => 1
}

In this way you'll be reciving logstash events every second with the content of your file into the message field (Monitoring)..... However this is just a event per file and your file content will be just in one line like : This\n\is\na\n\multiline\nmessage or something like that, what I did to fix this was to modify my exec.rb to split the line like:

loop do
  start = Time.now
  @logger.info? && @logger.info("Running exec", :command => @command)
  out = IO.popen(@command)
  @pipe = IO.popen(@command, mode = "r")
  
  @pipe.each do |line|
    line = line.chomp
  # out.read will block until the process finishes.
    @codec.decode(line) do |event|
      decorate(event)
      event["host"] = hostname
      event["command"] = @command
      queue << event
    end  
  end
  out.close

  duration = Time.now - start
  @logger.info? && @logger.info("Command completed", :command => @command,
                                :duration => duration)

  # Sleep for the remainder of the interval, or 0 if the duration ran
  # longer than the interval.
  sleeptime = [0, @interval - duration].max
  if sleeptime == 0
    @logger.warn("Execution ran longer than the interval. Skipping sleep.",
                 :command => @command, :duration => duration,
                 :interval => @interval)
  else
    sleep(sleeptime)
  end
end # loop

taking the idea from pipe input plugin ;). This will split your message and will send an event per every line in your file...
Not sure how could this affect in your performance however.
Hope this help

8

Thanks Rdoto - pretty awesome. Probably too much for what we need. Just need to show the latest set of config options for a handful of key properties and allow view/search/filtering. thank you for your suggestion though.

9

Thanks for the recommendations. I think we are opting for a simpler solution: setup a python script that will detect change in one of these property files and when that happens, the script will then also copy that property file to the directory that logstash monitors, logstash does its stuff and voila!