Logstash - Multiple Logs

Elastic Stack Logstash
1

Hi,

I'm just starting to play with ELK, we're looking at shipping different logs with different filters from each server to a centralised place.

filebeat config;

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/gwlogs/ContactManager/logs/ablog.log
    • /var/log/gwlogs/PolicyCenter/logs/pclog.log
    • /var/log/gwlogs/PolicyCenterAggs/logs/pclog.log
    • /var/log/gwlogs/BillingCenter/logs/bclog.log
      tags: ["guidewire"]

    multiline.pattern: '(^\s|^([A-Za-z])[A-Za-z0-9_].([A-Za-z])[A-Za-z0-9_].)'
    multiline.match: after

  • type: log

    paths:

    • /var/log/gwlogs/ContactManager/logs/AggregatorLog.log
    • /var/log/gwlogs/PolicyCenter/logs/AggregatorLog.log
    • /var/log/gwlogs/PolicyCenterAggs/logs/AggregatorLog.log
    • /var/log/gwlogs/BillingCenter/logs/AggregatorLog.log
      tags: ["aggregator"]

output.logstash:

The Logstash hosts

hosts: ["10.60.154.69:5044"]

Logstash config;

input {
beats {
port => 5044
}
}

filter {
if "guidewire" in [tags] {
grok {
match => {
"message" => "(?([a-z]{3}[0-9]{2}[a-z]{2}[0-9]{2}){0,1})\s+(?([A-Za-z0-9.@-]*){0,1})\s+(%{TIMESTAMP_ISO8601:logdate})?\s+(%{LOGLEVEL:loglevel})\s+%{GREEDYDATA:body}"
}
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
tag_on_failure => [ "_not_interested" ]
}
if "_not_interested" in [tags] {
drop { }
}
else if "aggregator" in [tags] {
grok {
match => { "message" => "%{GREEDYDATA:body}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
}
}

output {
if "guidewire" in [tags] {
hosts => ["http://localhost:9200"]
index => "%{[@metadata]]beat]}-%{+YYYY.MM.dd}"
}
if "aggregator" in [tags] {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}

Doesn't seem to work, any help would be appreciated.

Thanks,

Chris

2

Anyone?

3

Please edit your post and format it using markdown. Also, you have not given any indication of how it does not work. You cannot expect us to guess.

4

Hi,

Sorry - I can't see a way to edit the original post?

I'd like it to take the first batch of the logs and process them using the pattern before passing them to logstash, which seems to work fine in isolation. When I add the second log paths in it seems to ignore them - so was wondering on the correct syntax to add multiple log paths in and process different logs..

Thanks,

Chris

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.