Output Logs In Original Form


(Ernest Mueller) #1

Hey all, I'm a new logstash user. I'm trying to ease in a client to ELK who is currently analyzing their logs using an existing tool. I wanted to start by just using logstash for log shipping and aggregating them to one server, and then add on the EK portion to show their value later without disrupting their current process.

(Difficulty: Windows)

So I set up the server and logstash-forwarder and have both of them running, and extracting just the messages from the logs on the server end. But what I can't figure out how to do is to split them back out into their original filenames instead of one jumbo file with all the messages in it.

Desired goal - from a bunch of servers (hostname1...hostnameN), pull a bunch of log files and put them on the logstash server with date rotation. So hostname1:C:\file1.log becomes server:C:\logs\hostname1\file1.log.2015-07-10 et al.

Here's my logstash-forwarder.conf

{
"network": {
    "servers": [
        "localhost:12345"
    ],
    "ssl certificate": "C:/logstash-1.5.1/logstash.crt",
    "ssl key": "C:/logstash-1.5.1/logstash.key",
    "ssl ca": "C:/logstash-1.5.1/logstash.crt",
    "timeout": 15
},
"files": [
    {
        "paths": [
            "C:/Program Files/uptime software/uptime/logs/*"
        ]
    }
  ]
}

And my logstash.conf

input {
  lumberjack {
    # The port to listen on
    port => 12345
    ssl_certificate => "C:/logstash-1.5.1/logstash.crt"
    ssl_key => "C:/logstash-1.5.1/logstash.key"
	
    # Set this to whatever you want.
    type => "remotelogs"
  }
}
output {
  file {
    path => "C:/logstash-1.5.1/logs/%{host}/log.%{file}.%{+yyyy.MM.dd.HH}"
	message_format => "%{message}"
  }
}

This gives me a mkdir error when I try to start the logstash server. I've fooled around with different options for the path and it'll put all the lines in one file fine, I just want to end up with the original filenames (though ideally with date rotation).

Thanks!

Ernest


(Ernest Mueller) #2

I have gotten a little closer - the problem was the Windows path being passed along in %{file}, e.g. "file":"C:\Program Files\my software\foo\logs\thirdparty.log" was causing logstash server to choke and die with a mkdir exception.

I have kinda corrected it by using mutate to split off the C: thus:

input {
  lumberjack {
    # The port to listen on
    port => 12345
    ssl_certificate => "C:/logstash-1.5.1/logstash.crt"
    ssl_key => "C:/logstash-1.5.1/logstash.key"
	
    # Set this to whatever you want.
    type => "remotelogs"
  }
}
filter {
  mutate {
    split => { "file" => ":" }
	}
}
output {
  file {
    path => "C:/logstash-1.5.1/logs/%{host}/%{file[1]}.%{+yyyy.MM.dd.HH}"
	message_format => "%{message}"
  }
}

But it's clumsy, and if it gets a linux forwarder then there's no file[1]. I can't figure out how to do an "if" kind of thing. Ideas?


(Ernest Mueller) #3

OK I figured out how to make this work across Windows and Linux with help from a friend who's an experienced logstash user, so posting here for anyone that comes and looks in the future.

Pure log shipping, multiplatform. logstash.conf:

input {
  lumberjack {
    port => 12345
    ssl_certificate => "C:/logstash-1.5.1/logstash.crt"
    ssl_key => "C:/logstash-1.5.1/logstash.key"
  }
}
filter {
  if [type] =~ "windows" {
    mutate {
      split => { "file" => ":" }
	  }
	}
		
	if [type] =~ "windows" {
    mutate {
	  replace => { "file" => "%{file[1]}" }
	  }
	}
}
output {
  file {
    path => "C:/logstash-1.5.1/logs/%{host}/%{file}.%{+yyyy.MM.dd.HH}"
	message_format => "%{message}"
  }
}

logstash-forwarder.conf:

{
    "network": {
        "servers": [
            "localhost:12345"
        ],
        "ssl certificate": "C:/logstash-1.5.1/logstash.crt",
        "ssl key": "C:/logstash-1.5.1/logstash.key",
        "ssl ca": "C:/logstash-1.5.1/logstash.crt",
        "timeout": 15
    },
    "files": [
        {
            "paths": [
                "C:/Program Files/uptime software/uptime/logs/*"
            ],
			"fields": { "type": "windows" }
        }
    ]
}

Basically just set type: windows or something else on the forwarder, check for it and if it's there strip off the drive letter on the file name.


(system) #4