How to different syslog in the same port?

Elastic Stack Logstash
1

Hi all, I have many network devices, I forward their syslog to a logstash server, and then to elastisearch.
In logstash, I start many ports, such 514,515,516... and each port to different log types, and when output to es, one type to one index, so I can make index pattern in kibana in a easy way.

Here is the problem: some old devices can only forward syslog to port 514 than any other port. That means in the default port, many different type of devices in a same index.

So, I want to know, how to solve this? My purpose is to split different device's syslog so I can use different grok pattern in the logstash filter.

Thanks.

2

I would look at your data and see if there is any unique features you can use to separate them and then use conditionals to access them.

Maybe log.file.path?

3

well, I looked this syslog, the ip addr may be useful.

a log in kibana like this

{
  "_index": "syslog-generic-20220123",
  "_type": "_doc",
  "_id": "fzAZh34BidnVrFlllwQe",
  "_version": 1,
  "_score": 1,
  "_source": {
    "facility": 0,
    "@timestamp": "2022-01-23T13:21:48.215Z",
    "host": "1.2.3.4",
    "facility_label": "kernel",
    "type": "syslog-generic",
    "priority": 0,
    "message": "<134>dnscache: 1642944108 2022-01-23 21:21:48 queries:litedev.ys7.com IN A from client 10.150.0.134#60923,dns_server 1.2.3.4:53 responds to:112.17.34.109,about line:2,set MARK:0x2010403\n",
    "severity": 0,
    "severity_label": "Emergency",
    "tags": [
      "_grokparsefailure_sysloginput"
    ],
    "@version": "1"
  },
  "fields": {
    "severity": [
      0
    ],
    "tags.keyword": [
      "_grokparsefailure_sysloginput"
    ],
    "@version.keyword": [
      "1"
    ],
    "severity_label.keyword": [
      "Emergency"
    ],
    "type": [
      "syslog-generic"
    ],
    "message": [
      "<134>dnscache: 1642944108 2022-01-23 21:21:48 queries:litedev.ys7.com IN A from client 10.150.0.134#60923,dns_server 1.2.3.4:53 responds to:112.17.34.109,about line:2,set MARK:0x2010403\n"
    ],
    "priority": [
      0
    ],
    "tags": [
      "_grokparsefailure_sysloginput"
    ],
    "@timestamp": [
      "2022-01-23T13:21:48.215Z"
    ],
    "type.keyword": [
      "syslog-generic"
    ],
    "message.keyword": [
      "<134>dnscache: 1642944108 2022-01-23 21:21:48 queries:litedev.ys7.com IN A from client 10.150.0.134#60923,dns_server 1.2.3.4:53 responds to:112.17.34.109,about line:2,set MARK:0x2010403\n"
    ],
    "host": [
      "1.2.3.4"
    ],
    "@version": [
      "1"
    ],
    "host.keyword": [
      "1.2.3.4"
    ],
    "facility": [
      0
    ],
    "facility_label.keyword": [
      "kernel"
    ],
    "severity_label": [
      "Emergency"
    ],
    "facility_label": [
      "kernel"
    ]
  }
}

in the json log, "1.2.3.4"is the unique source of the device.

So, the probelm becomes, how to ideneitify the ip addr in logstash filter?

I tried this

input{
    syslog{
        type => "syslog-generic"
        host => "ip-addr-of-logstash"
        port => 514
    }
}

filter {

 if ([fields][host] == "1.2.3.4") {
    mutate {
      replace => {
        "[type]" => "syslog-dns-query"
      }
    }
  }

}

output {


    if [type] == "syslog-dns-query" {
       elasticsearch {
        hosts => ["myes-addr"]
            user => "elastic"
            password => "changeme"
            index => "syslog-dns-query-%{+YYYYMMdd}"
        }
    }
}

But no index named syslog-dns* appears in kibana.
So, anything wrong?

Thank you .

4

Your host is an array. So you need to access the first element in the array. Be aware since it's an array it might change and add more hosts in there. I would verify you are only ever going to get 1 in order for this to work.

if ([fields][host][0] == "1.2.3.4") {

5

Thank you.

Well, I tried, seems didn't work...

I noticed the host is a text, like this:

image
image1253×1039 53.4 KB

So can I use this in filter?

input{
    syslog{
        type => "syslog-generic"
        host => "ip-addr-of-logstash"
        port => 514
    }
}

filter {
 if ([_source][host] == "1.2.3.4") {
    mutate {
      replace => { "[type]" => "syslog-dns-query"}
    }
  }
}

At first, the syslog from port 514 get a type of "syslog-generic" in input section, and then in filter section we changed the type to "syslog-dns-query" where _source host is "1.2.3.4".

unfortunately it doesn't work...I can not find any index named syslog-dns-query in my kibana.

6

If it's field host then you would just do this. Don't need the _source.

if [host] == "1.2.3.4" {
7

well...I made a stupid mistake...

The right filter is
if [host] == "1.2.3.4" {
and I uesed this:
if ([host] == "1.2.3.4") {
now, it works fine.

The filter section now this:

input{
    syslog{
        type => "syslog-generic"
        host => "ip-addr-of-logstash"
        port => 514
    }
}

filter {
 if [host] == "1.2.3.4" {
    mutate {
      replace => { "[type]" => "syslog-dns-query"}
    }
  }
}

Thanks for help!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.