Make logstash send logs via lumberjack to logstash

Elastic Stack Logstash
1

I'd like to send logs, jmx data and collectd data to a central elasticsearch server from several clients. I've setup collectd on the client to send data to LS running on the ES server and that works just fine. I can setup logstash-forwarder to send logs to LS on the ES server and that works just fine. Haven't tried it yet but I know to send jmx data, I need to use LS to send to LS on ES so I've tried to configure LS instead of logstash-forwarder to send the logs. I'm able to get logs but can't figure out how to setup the config so I'm able to filter them. lumberjack on sent by LS as opposed to logstash-forwarder doesn't seem to be as rich? How can I even set the type for several different log files so I can parse / filter them? I can set the type when I receive them in LS but they then all get the same type (I can't differentiate between say the apache access and apache error logs).

Or is there a better way?

1 Like
2

You can send each to a different port and assign a type there or you can use grok to match a specific pattern for each and then apply a type.

3

but not something simple like on the client logstash-forwarder config:

{
  "paths": [
    "/var/log/apache2/access.log"
  ],

  "fields": { "type": "apache-access" }
}

then on the server:

filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}

4

Yes, you can do that. Check the docs for more.

5

Yes, you can do that with the logstash-forwarder but I want to do it with the lumberjack output plugin for logstash - I don't want to run both on the same client. I need to run logstash if I want to send more than just logs to ES (jmx data for example).

6

Set the type at the origin, i.e. in the input plugin:

input {
  path => "/var/log/apache2/access.log"
  type => "apache-access"
}
7

I tried that but the type didn't come thru once it got to the other side. I'm probably missing something fundamental... Once I get to the office, I'll include all the details of what happens when I tried that and what my config looked like.

8

Wild guess: You're using the plain codec. Use the json codec at both ends instead.

9

Yup - let me try that... I'll get back once I get to the office and try...

10

Thank you thank you that's just what I needed / was missing. Now I can play with the filters and get what I want :smile: Much appreciated!